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ABSTRACT 


The  objectives  of  this  thesis  were  to  validate  and 
evaluate  the  Baroudi  and  Orlikowski  [2]  short-form  UIS 
instrument  as  a  surrogate  measurement  of  effectiveness  of  the 
Department  of  Defense's  Composite  Health  Care  System  (CHCS) . 
Hurd  [1],  using  the  short-form  UIS  instrument  at  the  Naval 
Hospital  Charleston,  suggested  that  a  fundamental  change  in 
the  instrument's  structure  due  to  significant  impact  of 
contractor's  services  with  the  CHCS.  The  three  Navy  CHCS 
operational  testing  and  evaluation  hospitals  were  used  in  this 
study . 

The  short-form  UIS  instrument  was  found  to  be  inadequate 
for  evaluation  of  overall  UIS  of  the  CHCS.  Limitations  found 
were:  a  fundamental  change  in  the  factor  structure;  unevenly 
distributed  variables  in  a  couple  of  factors;  low  reliability 
in  one  factor;  no  assessment  of  user  interface  with  the 
system;  and  inconsistent  convergent  validity  findings.  A 
proposed  re-design  is  presented. 

Statistically  significant  differences  in  UIS  between 
groups  and  sites  were  found  for  the  four  factor  solution. 
Participants  were  divided  into  three  functional  user  groups: 
Physicians,  Medical  Support,  and  Administrative  Support.  All 
three  groups  were  satisfied  with  the  local  Management 
Information  Department  staff  and  services  and  the  system's 
output.  All  three  groups  were  dissatisfied  with  the 
contractor  services.  Physicians  were  significantly  less 
satisfied  in  three  out  of  the  four  factors.  No  significant 
correlation  between  time  of  use  of  the  system  and  UIS  was 
found,  except  for  the  Administrative  group's  positive 
correlation  with  the  system's  output. 
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I .  INTRODUCTION 


Modern  health  care  is  becoming  ever  increasingly  complex 
and  costly  in  its  delivery.  The  Department  of  Defense  in  its 
effort  to  combat  these  issues  has  put  considerable  effort  and 
time  into  health  care  computer-based  information  systems. 
The  latest  system  development  underway  is  the  Composite 
Health  Care  System  (CHCS) .  This  system  integrates  various 
health  care  components  within  the  hospital-based  health  care 
catchment  area. 

Good  management  practices  deem  assessment  of  effectiveness 
of  any  implementation  effort.  Effectiveness  of  a  computer- 
based  information  system  can  be  defined  as  meeting  the  needs 
of  the  users  it  was  intended  to  help.  A  commonly  used 
assessment  of  system  effectiveness  used  in  the  literature  is 
the  surrogate  measure  of  user  information  satisfaction  (UIS) — 
an  empirical  psychometric  approach.  Hurd  [1]  conducted  an 
evaluation  of  UIS  at  the  Naval  Hospital  Charleston,  South 
Carolina — one  of  the  Navy's  three  naval  hospital  operational 
testing  and  evaluation  (OT&E)  sites  for  CHCS.  In  his 
research,  he  used  the  UIS  instrument  and  evaluation 
methodology  developed  and  validated  by  Baroudi  and  Orlikowski 
[2].  The  Baroudi  and  Orlikowski  UIS  instrument  explained 
overall  satisfaction  to  be  the  summation  of  each  item 
(question)  score  which  is  the  composition  of  three  factors. 
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According  to  Baroudi  and  Orlikowski  [2],  the  three  factors 
that  make  up  overall  UIS  are:  1)  Electronic  Data  Processing 
(Management  Information  Department  in  naval  hospitals)  Staff 
and  Services,  2)  Information  Product  Output,  and  3)  User 
Knowledge  and  Involvement  of  the  system.  Hurd  [1]  separated 
the  one  factor  dealing  with  Management  Information  Department 
staff  and  services  into  essentially  two  factors  due  to  the 
fact  that  two  of  the  questions  dealt  with  services  not 
provided  internally — rather  they  are  performed  by  an  external 
vendor.  This  resulted  in  a  four  factor  instrument,  one  that 
demonstrated  che  additional  impact  of  contractor  services  on 
overall  UIS  in  this  setting.  This  fundamental  change  in  the 
instrument's  structure  would  tend  to  suggest  invalidation  of 
the  overall  UIS  score,  especially  since  the  original 
instrument  was  specifically  designed  (content  of  the 
questions)  to  support  the  three  factors  mentioned  above  [2]. 

A.  RESEARCH  OBJECTIVES 

The  primary  objective  of  this  research  is  to  evaluate  and 
validate  the  Baroudi  and  Orlikowski  UIS  instrument  (to  be 
referred  to  as  the  short-form  UIS  instrument  throughout  this 
paper)  using  a  particular  setting  and  system — the  CHCS  at 
naval  hospitals.  Hurd's  data  will  be  included  with  the  data 
collection  samplings  from  the  two  other  naval  hospitals 
designated  as  CHCS  OT&E  sites.  Questions  to  be  answered  are: 
1)  does  the  four  factor  (Management  Information  Department 
Staff  and  Services,  Contractor  Services,  Information  Product 
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Output,  and  User  Knowledge  and  Involvement)  exist  using 
common  factor-analytic  techniques?;  2)  is  there  equivalence 
(or  invariance)  of  the  four  factor  instrument  across  the 
three  Navy  CHCS  OT&E  sites?;  and  3)  using  follow-up  random 
interviews  of  system  users,  literature  review,  and  empirical 
judgment,  does  the  short-form  UIS  instrument  adequately  assess 
overall  satisfaction? 

Data  collected  from  this  research  can  be  of  additional 
informational  value.  Hurd's  [1]  findings  indicated  that  1) 
physicians  were  least  satisfied  with  training;  2)  physicians 
were  least  satisfied  with  understanding  the  system;  3)  that 
all  his  designated  user  groups  (physicians,  ancillary 
support,  and  administration)  were  dissatisfied  with  the 
services  provided  by  the  contractor  (i.e.,  handling  and  speed 
of  implementation  of  system  changes) ;  and  4)  there  was  no 
correlation  between  the  length  of  time  of  system  use  and 
overall  UIS  score  for  all  user  groups. 

Secondary  objectives  will  be  to  evaluate  the  following:  1) 
do  physicians  significantly  stand  out  as  less  satisfied  as 
compared  with  other  user  groups?;  2)  within  user  groups  are 
there  significant  differences  between  location  sites?;  and  3) 
is  there  a  correlation  between  length  of  time  of  use  of  the 
CHCS  system  and  UIS? 

B.  OVERVIEW 

Chapter  II,  discusses  a  brief  review  of  the  alternative 
measures  of  system  effectiveness,  some  of  the  problems 
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associated  with  purely  quantitative  measures  (i.e.,  economic 
analyses) ,  and  the  development  of  the  attitude  psychometric 
instrument  that  is  used  in  this  study.  Chapter  III, 
discusses  the  research  methodology  used  for  the  primary  and 
secondary  objectives,  including  a  summary  of  the  CHCS  and  the 
setting  for  this  study.  Chapter  IV,  presents  the  findings 
from  the  data  collection,  tabulation,  and  testing  techniques 
employed.  Chapter  V,  discusses  the  significant  results  and 
their  impact  towards  the  primary  and  secondary  objectives. 
Lastly,  Chapter  VI,  presents  the  conclusions  derived  from 
this  study,  as  well  as  those  issues  for  future  research. 


4 


II.  USER  INFORMATION  SATISFACTION  AND  ITS  MEASUREMENT 

A.  INTRODUCTION 

Computer-based  information  systems  (CBIS)  can  be  very 
expensive.  As  such,  administrators  and  researchers  have  long 
attempted  to  identify  a  meaningful  means  to  evaluate  CBIS 
effectiveness.  The  most  common  approach  suggested  by 
Information  Systems  (IS)  literature  for  measuring  CBIS 
effectiveness  has  been  user  information  satisfaction  (UIS) . 
Ives  et  al.,  define  UIS  as  "the  extent  to  which  users  believe 
the  IS  available  to  them  meets  their  information  requirements 
[3,  p.785]."  UIS  is  a  perceptual  or  subjective  "surrogate" 
measure  of  system  effectiveness.  Objective,  quantitative 
measures,  using  economic  analysis  (i.e.,  cost-benefit 
determinations)  have  been  reported  to  present  problems  in 
quantifying  the  actual  benefits  of  a  CBIS. 

B.  METHODS  TO  ASSESS  EFFECTIVENESS 

Generally  speaking,  measurement  of  CBIS  effectiveness 
falls  into  two  categories:  economic  and  psychometric. 

Ideally,  the  CBIS's  effectiveness  or  value  to  the 
organization  would  be  evaluated  using  an  objective  economic 
analysis.  The  costs  associated  with  the  system  operation 
would  be  subtracted  from  the  actual  benefits  from  using  the 
system  equated  in  monetary  terms  resulting  in  the  net 
monetary  valuation  of  effectiveness  to  the  organization. 
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However,  Ives  et  al.,  and  many  other  researchers  have  argued 
that  this  is  not  a  easy  task  due  to:  1)  difficulty  to 
recognize  and  quantify  intangible  benefits;  2)  certain 
benefits  such  as  decision  support  ad  hoc  queries  are 
virtually  impossible  to  quantify;  and  3)  even  when 
organization's  attempt  to  quantify  these  items,  they  are 
usually  highly  subjective,  and  undocumented  in  how  derived, or 
unavailable  for  research  [1-10].  Approaches  that  have  been 
advocated  for  measuring  and  improving  CBIS  effectiveness 
include  usage  estimation  [11],  incremental  performance  in 
decision  making  effectiveness  [12],  cost-benefit  analysis 
[13],  information  economics  [14],  utility  analysis  [15],  the 
analytic  hierarchy  approach  [16],  information  attribute 
examination  [17],  and  user  information  satisfaction. 

A  number  of  researchers  have  studied  UIS  as  the  measure  of 
CBIS  effectiveness  [1-10]  and  [18-25].  Powers  and  Dickson 
[7],  and  Cerullo  [25]  suggest  that  user  satisfaction  (a 
psychometric  measure)  is  the  most  important  element  in 
determining  CBIS  success.  According  to  Conrath  and  Mignen 
[20],  the  key  contribution  in  developing  a  tool  for  measuring 
and  analyzing  computer  user  satisfaction  has  been  the 
research  work  of  Bailey  and  Pearson. 

C.  DEVELOPMENT  OP  THE  DIB  QUESTIONNAIRE  INSTRUMENT 

Bailey  and  Pearson's  [18]  research  involved  a  literature 
review  of  22  studies  of  the  computer/interface  to  establish  an 
initial  list  of  variables  that  might  affect  user  satisfaction. 
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From  the  literature  review,  they  identified  36  variables. 
Next,  they  had  three  data  processing  professionals  review  the 
list,  wherein  the  list  was  expanded  to  38  variables.  The  38 
variable  list  was  empirically  compared  to  interview  responses 
from  32  managers  of  information  systems  in  eight  different 
organizations.  After  this  interview  process,  one  additional 
variable  was  added  to  the  list  for  a  total  of  39  variables. 
Using  the  bidirectional  model  of  Wanous  and  Lawler  [26]  which 
suggests  that  satisfaction  is  the  sum  of  one's  positive  and 
negative  reactions  to  a  set  of  variables,  and  using 
semantic  differential  technique  [27],  Bailey  and  Pearson 
added  dimensionality  and  intensity  of  an  individual's 
reaction  to  each  variable  based  on  the  use  of  adjectives  to 
explain  the  user's  perceptions.  A  seven  interval  Likert-type 
measurement  scaling  model  was  used  to  enumerate  the  adverbial 
qualifiers:  extremely,  quite,  slightly,  neither/equally, 
slightly,  quite,  and  extremely  for  each  of  the  negative  to 
positive  (bipolar)  adjective  pair.  Each  variable  had  four 
adjective  pairs,  one  pair  to  rate  satisfactory  versus 
unsatisfactory,  and  one  pair  to  rate  whether  the  variable  was 
important  versus  unimportant.  The  previously  interviewed  32 
managers  were  asked  to  fill  out  the  questionnaire,  with  29 
being  completed  and  returned.  Although  the  sample  size  was 
small,  Bailey  and  Pearson's  instrument  presented  significant 
progress  toward  the  development  of  a  standard  measure  of 
UIS. [18] 
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Bailey  and  Pearson's  [18]  research  instrument  or  construct 
for  measuring  UIS  using  their  39  variable  questionnaire,  set 
the  stage  for  further  research.  They  recommended  further 
research  and  validation  efforts  in  a  wide  variety  of  user 
environments,  and  the  application  of  factor  analysis  to  see 
if  and  when  the  list  of  variables  can  be  reduced. 

Ives,  Olson,  and  Baroudi  [3],  chose  to  undertake  a 
replication  of  Bailey  and  Pearson's  findings  to  reinforce  the 
validity  of  the  instrument  through  further  tests,  and  reduce 
the  length  of  the  instrument  in  an  attempt  to  produce  a 
standard  "short-form"  instrument.  Their  research  was  based 
on  the  completed  questionnaires  from  200  production  managers 
in  U.S.  manufacturing  organizations  selected  from  a 
commercially  obtained  mailing  list.  From  their  research, 
the  list  of  variables  from  the  original  Bailey  and  Pearson 
instrument  were  reduced  to  22,  and  the  four  bipolar  adjective 
pairs  reduced  to  two  pairs.  Factor  analysis  testing  was 
conducted  on  the  22  variable  instrument  to  see  if  the 
information  could  be  reduced  into  fewer  latent  variables 
(unobserved  variables)  or  factors.  Essentially,  four  factors 
were  identified:  Electronic  Data  Processing  (EDP)  Staff  and 
Services,  Information  Product  Output,  User  Knowledge  and 
Involvement,  and  Vendor  Support.  Vendor  Support  was  dropped 
due  to  only  one  variable  loading  into  this  factor,  and  the 
researcher's  belief  at  the  time  that  it  was  not  a  significant 
component  to  overall  user  information  satisfaction 


8 


assessment.  Their  reliability,  validity,  and  correlation  data 
provided  substantial  evidence  that  the  shortened  form 
questionnaire  is  a  sound  general  measure  of  Bailey  and 
Pearson's  original  UIS  concept. 

Ives  et  al.  [3],  recommended  further  research  to  validate, 
extend,  and  disseminate  the  instrument.  Additionally,  they 
recommended  a  change  in  the  instrument  format.  The  original 
Bailey  and  Pearson  [18]  instrument,  and  the  Ives,  Olson,  and 
Baroudi  [3]  instrument  had  each  of  the  bipolar  adjective  pairs 
in  the  same  directional  format,  scored  negatively  to  the 
right,  and  scored  positively  to  the  left.  Reliability  would 
be  further  increased  by  mixing  up  the  direction  of  some  of 
the  adjective  pairs  for  some  of  the  variables  [3]. 

Baroudi  and  Orlikowski  [2]  further  refined  the  short  form 
of  the  UIS  instrument  previously  researched  by  Ives  et  al.  [3] 
into  a  13  question  (variable)  measure  of  UIS.  The  13 
variables  were  selected  because  they  believed  the  selected 
variables  displayed  the  most  desirable  psychometric  properties 
for  the  three  UIS  factors  (EDP  Staff  and  Services,  Information 
Product  Output,  and  User  Knowledge  and  Involvement) .  One  of 
the  objectives  of  their  study  was  to  develop  a  standard 
"short-form"  UIS  instrument.  As  with  the  previous 
instruments,  each  bipolar  adjective  pair  is  scored  using  a 
seven  interval  Likert-type  measurement  scaling  model  with 
values  ranging  from  -3  (extremely  dissatisfied)  to  +3 
(extremely  satisfied) ,  and  zero  indicating  a  neutral 


response.  Each  variable  (question)  is  scored  by  taking  the 
average  of  the  two  bipolar  adjective  pair  scores.  They  mixed 
up  the  direction  of  a  number  of  the  adjective  pairs  as 
recommended  from  previous  research. 

Baroudi  and  Orlikowski  [2]  specifically  picked  the 
variables  in  the  short-form  (Appendix  A,  Part  B)  that 
measured  the  three  factors  that  Ives,  Olson,  and  Baroudi  [3] 
found  to  comprise  overall  UIS.  Three  subtotals  representing 
the  three  factors  were  calculated  by  averaging  the  variables 
(questions)  that  comprised  each  (ranging  from  -3  to  +3)  . 
Questions  1,  2,  6,  11,  and  12  factor  loaded  in  the  assessment 
of  the  attitude  and  responsiveness  of  the  EDP  staff  as  well  as 
the  quality  of  the  relationship  between  the  user  and  the  EDP 
staff  for  scoring  the  first  factor.  Questions  7,  8,  9,  10, 
and  13  factor  loaded  in  the  assessment  of  the  quality  of 
output  delivered  by  the  information  system  for  the  scoring  of 
the  second  factor.  Questions  3,  4,  and  5  factor  loaded  in 
the  assessment  of  the  quality  of  training  provided, 
understanding  of  the  system,  and  participation  in  the  system's 
development  for  scoring  the  third  factor.  The  overall  UIS 
score  was  determined  by  the  summation  of  each  of  the  13 
variable  (question)  scores.  The  total  score  could  range  from 
-39  to  +39. 

Data  was  collected  from  368  employees,  mostly  clerical  and 
support  personnel,  of  26  New  York  area  organizations  using 
transaction  processing  computer  systems.  Follow-up  interview 
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measure  of  UIS  from  five  of  the  organizations  was  conducted  in 
addition  to  internal  construct  validity  and  reliability 
analysis.  The  data  indicated  that  the  interview  assessments 
of  user  satisfaction  or  dissatisfaction  correlated  well  with 
the  satisfaction  scores  obtained  by  the  short-form  instrument 
and  provides  some  evidence  for  the  instrument's  convergent 
validity. 

Igbaria  and  Nachman  [5]  provided  further  validation  of  the 
Baroudi  and  Orlikowski  short-form  instrument  in  their  study 
using  a  different  sample  of  users.  They  performed  a  second 
order  factor  analysis  to  test  for  underlying  homogeneity  in 
the  first  order  three  factors  that  were  extracted  to  assess 
overall  UIS.  Their  results  confirmed  the  homogeneity  of  the 
items  and  the  appropriateness  of  combining  the  13  questions 
as  an  overall  measure  of  UIS.  Additionally,  they  added  a 
demographic  section  to  the  short-form  instrument  to  assess 
the  role  that  individual  characteristics  had  with  UIS.  Based 
on  previous  research,  they  included  computer  experience, 
education,  age,  gender,  organizational  level,  and  time  of  use 
of  computer  system  as  user  variables  for  UIS.  From  their 
research,  they  found  that  education,  organizational  level  and 
gender  were  not  correlated  with  UIS.  However,  their  findings 
indicated  a  significant  positive  relationship  between  UIS  and 
length  of  time  using  the  CBIS. 

Hurd  [1]  used  the  Baroudi  and  Orlikowski  [2]  short-form 
UIS  instrument  in  his  study  of  a  specific  CBIS  (the  Composite 
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Health  Care  System)  at  a  particular  site,  the  Naval  Hospital 
Charleston,  South  Carolina.  Hurd  also  used  a  demographic 
section  with  essentially  the  same  salient  user  variables  as 
Igbaria  and  Nachman  [5],  modified  to  meet  the  site  under 
study  (Appendix  A,  Part  A).  Hurd's  findings  suggested  no 
correlation  of  overall  UIS  with  time  of  system  use.  However, 
looking  at  the  three  groups  of  users  (physicians,  ancillary 
personnel,  and  administrators),  Hurd  noted  that  there  were 
identifiable  trends.  Physician's  overall  UIS  tended  to 
remain  essentially  unchanged  with  time  of  system  use. 
Administrators  exhibited  a  positive  trend,  whereas  the 
ancillary  personnel  exhibited  a  negative  trend  between  UIS  and 
time  of  system  use. 

The  instrument  used  by  Hurd  (Appendix  A)  was  utilized  in 
this  study  to  extend  the  study  of  CHCS  user  satisfaction  at 
the  two  other  CHCS  OT&E  naval  hospitals  for  validation  of  the 
instrument,  comparison,  and  time  of  use  correlation  testing. 
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III.  RESEARCH  METHODOLOGY 


A.  INTRODUCTION 

This  study  has  two  main  objectives.  The  primary  objective 
is  to  evaluate  the  construct  validity  of  the  short-form  UIS 
instrument.  The  second  objective  is  a  follow  on  from  the 
primary  objective.  The  findings  from  the  currently  used 
short-form  UIS  instrument  will  be  presented  and  evaluated  in 
light  of  the  construct  validity  for  user  group/site  UIS 
comparisons.  In  addition,  the  correlation  between  the  length 
of  time  the  user  has  used  the  CHCS  and  UIS  will  be  evaluated. 
First,  the  sampling  setting  and  participants  will  be 
discussed.  This  will  also  include  a  brief  background  about 
the  CHCS.  Next,  the  measurement  scales  used  in  this  study  will 
be  presented  along  with  the  level  of  confidence  used  in  the 
statistical  analysis. 

Validation  of  the  short-form  UIS  instrument  will  take  a 
series  of  progressive  steps  in  order  to  answer  the  research 
questions.  Although  the  three  sampling  sites  are  relatively 
the  same  (i.e.,  medium-sized  naval  hospitals),  and  could  be 
assumed  to  represent  a  relatively  homogenous  sampling 
population,  oneway  analysis-of-variance  (ANOVA)  will  be 
performed  on  the  sampling  demographics.  Using  the  three  site 
combined  data,  common  factor  analysis  methods  will  be 
conducted.  Common  factor  analysis  is  considered  one  of  the 
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most  powerful  methods  of  construct  validation  as  it  allows 
the  examination  of  the  underlying  structure  of  the  overall 
measure  [3] [27].  Internal  consistency  or  reliability  testing 
of  each  factor  as  well  as  other  statistical  measures  of 
overall  factor  structure  "goodness  of  fit"  will  be  used. 
Equivalence  testing  is  conducted  to  test  the  invariance  of 
the  factor  structure  and  loading  across  the  three  sites,  and 
is  important  to  demonstrate  that  there  is  no  instrument  bias. 
Lastly,  responses  from  user  interviews,  and  comments  offered 
on  the  completed  instruments  will  be  used  to  look  at 
convergent  validity. 

Evaluation  of  the  findings  from  the  short-form  UIS 
instrument  as  mentioned  above  will  rely  on  the  outcome  from 
validation.  Presentation  of  the  data  will  use  statistical 
testing  techniques  to  demonstrate  differences  in  means 
between  the  defined  user  groups  and  sites.  Correlation 
testing  between  length  of  time  user  use  of  the  CHCS  and  UIS 
will  also  be  evaluated.  Correlation  statistical  methods  will 
be  used  and  tested  for  significance.  Additionally,  trend 
analysis  will  be  conducted  and  presented. 

B.  SAMPLE  AND  DATA  COLLECTION 

1.  Background  of  the  Composite  Health  Care  System  (CHCS) 

CHCS  is  a  state-of-the-art,  integrated,  medical 
information  system  the  Department  of  Defense  (DoD)  is  testing 
for  implementation  at  its  medical  treatment  facilities.  On 
the  leading  edge  of  technology  and  beyond  the  capabilities  of 


systems  commercially  available,  CHCS  is  designed  to  improve 
the  timeliness,  availability,  and  quality  of  patient-care 
data.  It  will  replace  manual  and  automated  information 
system  now  supporting  DoD  medical  treatment  facilities.  At 
individual  hospitals,  it  will  integrate  the  functional  work 
centers  of  inpatient  and  outpatient  care  facilities,  patient 
administration,  patient  appointment  and  scheduling,  nursing, 
laboratory,  pharmacy,  radiology,  and  clinical  dietetics. 
CHCS  is  intended  to  provide  physicians  with  immediate  access 
to  patient  medical  records.  [28] 

The  approval  of  the  Mission  Elements  Need  Statement 
(MENS)  in  February  1979  affirmed  the  DoD's  goal  of  providing 
integrated  information  support  via  a  computerized  system  for 
its  medical  treatment  facilities.  A  two-stage  competitive 
acquisition  process  was  undertaken  to  solicit  industry's  best 
solution  to  the  needs  of  DoD  medical  treatment  facilities. 
On  September  10,  1986,  stage  I  of  the  contract  was  awarded  to 
four  competing  vendors.  Each  vendor  was  to  design,  develop, 
and  implement  levels  I  and  II  of  CHCS  (essentially  outpatient 
services)  at  predesignated  test  sites.  Stage  II  consisted  of 
an  extensive  evaluation  that  included  an  extended  benchmark 
test  in  order  to  aid  in  the  selection  of  one  of  these 
vendors.  Based  on  the  evaluation,  Science  Applications 
International  Corporation  (SAIC)  of  San  Diego,  California  was 
se3ected  as  the  source  to  further  develop  and  deploy  CHCS  to 
12  operational  testing  and  evaluation  (OT&E  or  beta  test) 
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sites.  Of  the  12  OT&E  sites,  three  are  Navy,  five  are  Army, 
and  four  are  Air  Force  medical  treatment  facilities.  This 
group  of  OT&E  sites  represented  hospitals  with  the  number  of 
beds  ranging  from  40  to  886,  and  outpatient  clinic  visits 
ranging  from  247,285  to  1,573,369  per  year.  DoD  estimates 
CHCS  life-cycle  costs  (the  expected  life  of  the  system  is 
through  fiscal  year  2002)  for  full  deployment  to  767  medical 
treatment  facilities  at  $1.6  billion. [ 1] [28] [29] 

CHCS  is  a  menu-driven,  networked  system,  safeguarded 
via  a  password  access  hierarchal  assignment  according  to  the 
appropriate  level  of  communication  needs  of  the  user.  That  is 
to  say,  that  physicians  as  the  focal  point  of  health  care  and 
treatment  are  given  the  most  authority  to  input  and  extract 
information  from  the  system.  CHCS  also  provides  electronic 
mail  communication  and  the  system  is  intended  to  interface 
with  other  Hospital  System  Program  Office  (HSPO)  and  DoD 
initiatives  including:  food  service,  medical  logistics, 
Defense  Enrollment  Eligibility  Reporting  System  (DEERS) , 
Medical  Expense  and  Performance  Reporting  System  (MEPRS) , 
service-specific  administrative  systems,  tactical  automation 
systems,  and  National  Disaster  Medical  System  and  Veterans 
Administration  Systems. [29] 

2.  Sample  and  Data  Collection 

The  three  naval  hospital  OT&E  sites  (Charleston,  South 
Carolina;  Jacksonville,  Florida;  and  Camp  Lejeune,  North 
Carolina)  were  evaluated  in  this  study.  Hurd's  [1]  data  from 
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the  Naval  Hospital  Charleston,  South  Carolina,  was  used  in 
this  study  in  lieu  of  re-sampling.  The  three  sites  are 
essentially  the  same  size  (medium-sized  naval  hospitals)  with 
the  number  of  hospital  beds  ranging  from  170  to  184,  and 
outpatient  clinic  visits  ranging  from  360,000  to  about  570,000 
per  year  [28].  No  complete  inpatient  module  implementation 
had  occurred  at  any  of  these  three  naval  hospitals  at  the 
time  of  data  collection  for  study.  Therefore,  the  nursing 
unit  module  was  not  evaluated.  Additionally,  the  clinical 
dietetics  module  was  not  implemented.  The  modules  that  were 
implemented  were:  patient  administration  (PAD) ,  patient 
appointment  and  scheduling  (PAS) ,  pharmacy  (PHR) ,  laboratory 
(LAB) ,  and  radiology  (RAD) .  The  PAD  module  was  still  being 
run  in  parallel  with  the  AQCESS  system,  and  did  not  have  the 
cash  collections  component  (MSA)  on-line.  The  LAB  module  did 
not  have  the  blood  transfusion  service  component  on  line. 

The  short-form  UIS  instrument  used  was  the  same  as 
that  used  by  Hurd  [1]  due  to  the  incorporation  of  his  data 
from  the  Naval  Hospital  Charleston,  South  Carolina,  into  this 
study  (refer  to  Appendix  A)  .  However,  questions  8  and  10  were 
given  further  clarification  using  the  definitions  offered  in 
the  original  study  by  Bailey  and  Pearson  [18].  It  was  felt 
that  there  could  be  misunderstandings  or  ambiguity  in 
interpretation  by  the  subjects  without  this  clarification, 
and  would  not  jeopardize  the  instrument's  integrity  due  to 
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both  of  these  questions  are  expected  to  factor  within  the 
same  factor  (Information  Product  Output) . 

A  data  collection  set  consisted  of  a  cover  letter,  the 
short-form  UIS  instrument,  and  an  addressed  envelope.  The 
cover  letter  informed  the  subjects  that  their  responses  would 
be  treated  in  complete  confidentiality,  and  where  to  direct 
their  sealed  envelope  response  at  each  hospital.  The  Naval 
Hospital  Camp  Lejeune,  North  Carolina,  and  the  Naval  Hospital 
Jacksonville,  Florida,  were  the  two  other  Composite  Health 
Care  System  (CHCS)  facilities  sampled.  Prior  to  sending  out 
the  data  collection  sets  to  the  Naval  Hospital  Camp  Lejeune 
and  the  Naval  Hospital  Jacksonville,  each  facility's  CHCS 
Project  Officer  was  contacted  to  obtain  their  support  and 
assistance  in  this  study.  Additionally,  command  notification 
and  permission  was  obtained  at  each  facility. 

Each  CHCS  Project  officer  was  sent  250  data  collection 
sets.  Additionally,  general  guidance  for  dissemination  at 
each  facility  was  provided  to  each  CHCS  Project  Officer  to 
ensure  random  sampling  of  all  CHCS  users  in  the  outpatient 
areas  (i.e.,  physicians,  clinics,  laboratory,  radiology, 
pharmacy,  and  administrative  departments) .  Returned 
instruments  and  Hurd's  [1]  Naval  Hospital  Charleston  data 
were  coded  for  responses  and  categorized  into  three 
functional  groups:  physicians,  medical  support,  and 
administrative  support.  These  groupings  were  different 
from  Hurd's  [1]  study,  however,  tt-sse  groupings 
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better  represent  the  different  user  groups  in  terms  of  the 
function  provided  to  health  care,  and  therefore  use  of  the 
CHCS.  The  physician  group  is  self  explanatory.  The  medical 
support  group  consisted  of:  nurses,  pharmacists, 
technologists,  other  health  care  professionals,  technicians, 
and  hospital  corpsman  not  working  in  an  administrative 
department.  The  administrative  support  group  consisted  of: 
all  administrators,  secretarial  and  clerical  persons 
regardless  of  department  assigned  to,  and  all  other  persons 
regardless  of  profession  assigned  to  an  administrative 
department.  Education,  sex,  use  of  other  computer  systems, 
and  use  of  other  health  care  information  system  demographic 
information  were  given  numerical  values  (i.e.,  male=l, 
female=2,  no=0,  yes=l,  high  school  graduate=0,  etc.)  in  order 
to  allow  comparative  analysis  using  these  attributes,  or 
enable  ease  of  database  tabulation. 

All  responses  were  inputted  using  an  integrated 
modular  software  package  (Enable/OA) .  This  package  allows  the 
researcher  to  build  a  database  of  responses  and  ease  of  data 
retrieval  and  importation  of  this  data  into  a  spreadsheet 
format  for  statistical  analysis.  Each  of  the  questions 
(variables) ,  factors,  and  overall  satisfaction  scores  were 
computed  as  described  by  Baroudi  and  Orlikowski  [2]  (refer  to 
Chapter  II) . 
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C.  MEASUREMENT  SCALES 

In  psychological  or  behavioral  studies,  there  appears  to 
be  some  confusion  as  to  the  legitimacy  of  using  particular 
classes  of  mathematical  procedures  [8]  [27].  Specifically,  the 
use  of  parametric  statistical  procedures  verses  nonparametric 
procedures  with  measures  of  psychological  attributes. 
Parametric  statistical  procedures  have  more  statistical  power 
than  nonparametric  procedures,  yet  require  at  a  minimum  that 
there  exists  interval  scales.  According  to  Nunnally,  an 
interval  scale  is  "1)  the  rank-ordering  of  objects  is  known 
with  respect  to  an  attribute  and  2)  it  is  known  how  far  apart 
the  objects  are  from  another  with  respect  to  the  attribute, 
but  3)  no  information  is  available  about  the  absolute 
magnitude  of  the  attribute  for  any  object"  [27,  p.  16]. 
Scaling  models  such  as  the  seven  interval  Likert-type  scaling 
model  used  with  the  short-form  UIS  instrument,  are  applied  by 
the  researcher  to  what  appears  to  be  ordered  categories  (or 
ordinal  scales)  to  the  subjects,  to  convert  the  data  into 
interval  scales.  Nunnally  [27]  strongly  believes  that  it  is 
permissible  to  take  seriously  the  intervals  among  scores  in 
performing  analyses  of  attitude  such  as  that  used  with  the 
short-form  UIS  instrument.  It  is  beyond  the  scope  of  this 
paper  to  fully  explain  the  rationale  behind  these  arguments, 
and  it  is  recommended  that  readers  review  Nunnally  [27]. 

In  this  study,  parametric  procedures  will  be  used, 
however,  nonparametric  procedures  will  be  used  in  those  cases 


where  testing  failed  to  support  the  underlying  assumptions 
for  parametric  procedures  (specially,  in  the  oneway  ANOVA 
procedure) .  All  testing  was  conducted  at  a  confidence  level 
of  95%  or  alpha  =  0.05. 

D.  VALIDATION  OF  THE  INSTRUMENT 

1.  Homogeneity  of  the  Sample  Data  sets 

Oneway  analysis-of-variance  (ANOVA)  using  a  PC  version 
of  MINITAB  statistical  program  [30]  and  macro  programs 
provided  by  Zehna  [31]  were  performed  on  the  following 
demographic  attributes:  education,  age,  gender,  length  of 
time  (in  months)  of  CHCS  use,  use  of  other  computer  system, 
and  use  of  other  health  care  information  systems  to  ascertain 
homogeneity  of  the  sample  data  sets.  ANOVA  methods  have  been 
developed  to  test  for  differences  between  the  means  of 
several  groups.  In  this  study,  ANOVA  procedures  were  applied 
to  the  three  subpopulations:  Charleston,  Camp  Lejeune,  and 
Jacksonville.  Where  significant  differences  in  means 
occurred,  the  Scheffe  multiple  comparison  testing  was 
conducted  a  posteriori.  Additionally,  a  posteriori  testing 
for  normality  and  homogeneity  of  variance  was  conducted. 
Normality  was  tested  using  MINITAB ' s  option  for  computing  and 
storing  fitted  and  residual  values.  Applying  the  NSCORE 
function  to  compute  the  normal  scores  of  the  residuals  and 
then  compute  the  correlation  of  the  normal  score  with 
residuals  approximates  a  normal  distribution  if  the 
correlation  is  large  (i.e.,  the  closer  to  1.0  the  better) 
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[30].  The  Hartley's  Fmax  test  was  used  for  homogeneity  of 
variance. 

2.  Common  Factor  Analysis 

The  common  factor-analytic  model  is  different  from 
principal  components  analysis  in  that  it  makes  a  distinction 
between  common  and  specific  parts  of  variables.  In  principal 
components  analysis,  the  goal  is  to  construct  linear 
combinations  of  the  original  variables  that  account  for  a 
large  part  of  the  total  variation.  That  is  to  say,  the 
unobserved  factors  (latent  variables)  are  expressed  as 
functions  of  the  observable  variables,  and  is  variance 
oriented,  and  without  an  error  term.  The  common 

factor-analytic  model,  on  the  other  hand,  expresses  each 
observable  variable  in  terms  of  unobservable  common  factor  and 
a  unique  factor,  and  is  covariance  oriented.  The  common 
variance  of  a  variable  is  also  called  the  communality  of  the 
variable.  The  communality  of  a  variable  is  the  portion  of  a 
variable's  total  variance  that  is  accounted  for  by  the  common 
factors.  With  the  principal  components  analysis  there  is  no 
error  term.  Conceptually,  the  absence  of  an  error  term 
implies  that  the  observable  variables  are  measured  without 
error  and  that  the  unobservable  latent  principal  component  is 
a  perfect  linear  combination  of  its  measures  or  are  formative 
indicators  of  the  unobservable  factor.  Whereas,  common 
factor  analysis  is  reflective  in  that  the  indicators  subject 
to  measurement  error  are  a  function  of  unobservables. 
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Instrument  constructs  to  assess  attitude  are  typically  viewed 
as  underlying  factors  that  give  rise  to  something  that  is 
observed,  and  therefore  their  indicators  (i.e.,  the  observed 
variables)  should  be  viewed  as  reflective,  hence  the  use  of 
the  common  factor-analytic  model.  Common  factor-analytic 
techniques  can  better  serve  the  functions  of  searching  the 
data  for  qualitative  and  quantitative  distinctions  and, 
especially  testing  a  priori  hypotheses  and  statistical 
testing  criterion.  [32] 

The  maximum-likelihood  common  factor  analysis  is 
preferred  due  to  its  ability  to  test  hypotheses  about  the 
number  of  common  factors.  There  are  two  different  data 
analysis  contexts:  exploratory  and  confirmatory.  Exploratory 
factor  analysis  is  simply  searching  for  a  common  structure 
underlying  the  data  without  having  any  theoretical  hypothesis 
in  mind.  Whereas,  confirmatory  factor  analysis  there  exists 
some  prior  theoretical  information  on  the  common  structure 
underlying  the  data  and  one  wishes  to  confirm  or  negate  the 
hypothesized  structure. [32] [33] 

The  rotation  process  of  factor  analysis  pattern  matrix 
provides  a  clearer  delineation  of  the  pattern  of 
relationships.  Rotation  options  allow  for  a  simple  factor 
solution  to  become  clearer.  There  are  two  methods  in  which 
the  factor  axes  can  be  rotated.  Orthogonal  rotation 
preserves  the  original  orientation  between  the  factors  so 
that  they  are  still  perpendicular  after  rotation.  Whereas, 
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oblique  rotation,  the  factor  axes  can  be  rotated 
independently.  Varimax  orthogonal  rotation  is  one  of  the 
most  popular  rotation  techniques  used.  Varimax  rotation 
spreads  variance  evenly  among  factors  while  maintaining  the 
original  orientation  between  the  factors  so  that  they  are 
still  perpendicular  after  rotation.  The  procedure  seeks  to 
rotate  factors  so  that  the  variation  of  the  squared  factor 
loadings  for  a  given  factor  is  made  large. [32-34] 
a.  Exploratory  Factor  Analysis 

Initially,  exploratory  factor  analysis  was 
undertaken  using  the  SAS  maximum-likelihood  factor  analysis 
procedure  [33]  on  the  combined  data.  Multivariate  normality 
was  assumed  in  conducting  the  exploratory  factor  analysis. 
Cattell's  scree  test  was  performed  for  determining  the 
approximate  number  of  factors  to  extract.  The  Cattell's 
scree  test  is  simply  a  visual  determination  of  the  point 
where  the  factors  curve  above  an  approximate  straight  line 
made  from  the  bottom  roots  [34].  SAS  has  the  capability  of 
computing  the  Kaiser-Meyer-Olkin  (MSA  option)  measure  of 
sampling  adequacy  [33].  The  MSA  is  a  summary  of  the  extent 
to  which  the  variables  belong  together  and  are  thus 
appropriate  for  factor  analysis  [34].  MSA's  greater  than  0.8 
can  be  considered  good  [33] [34],  The  Schwartz's  Bayesian 
Criterion  is  used  to  determine  the  best  number  of  factors  to 
be  extracted  using  the  maximum  likelihood  factor  analysis 
procedure.  The  number  of  factors  that  yields  the  smallest 
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value  for  the  Schwartz's  Bayesian  Criterion  is  considered  the 
best  extraction  [33].  The  Schwartz's  Bayesian  Criterion 
according  to  the  SAS  user  guide  seems  to  be  less  inclined  to 
include  trivial  factors  than  either  the  Akaike's  Information 
Criterion  or  the  chi-square  test  [33].  In  the  literature, 
there  have  been  problems  reported  in  using  the  chi-square  test 
due  to  its  susceptibility  to  sample  size  [35-38].  SAS  also 
provides  the  Tucker  and  Lewis's  Reliability  Coefficient 
automatically  for  maximum-likelihood  factor  analysis 
procedure.  The  closer  the  Tucker  and  Lewis's  Reliability 
Coefficient  is  to  1.0  the  better  the  factor  solution  fit. 
SAS  also  automatically  computes  the  squared  canonical 
correlation  (which  is  the  same  as  squared  multiple 
correlations)  for  maximum-likelihood  factor  analysis 
procedure.  The  squared  multiple  correlation  (SMC)  for  each 
variable  is  the  relative  variance  in  that  variable  which  is 
accounted  for  by  the  overall  factor  solution  jointly  [33]  [39]. 
Basically,  the  SMC  represents  the  lower  bound  of  reliability 
that  each  variable  contributes  to  the  overall  factor 
structure.  The  maximum- likelihood  factor  analysis  procedure 
because  it  is  an  iterative  process  using  SMCs  for  initial 
estimates  (using  SAS)  is  susceptible  to  quasi-  or 
ultra-Heywood  cases.  It  is  beyond  the  scope  here  to  discuss 
these  anomalies,  however,  SAS  has  a  Heywood  option  which  sets 
to  1  any  communality  greater  than  1,  allowing  iterations  to 
proceed  until  convergence  criterion  is  met  [33].  The  Varimax 


rotation  option  was  used  in  the  SAS  procedure  program.  Lastly, 
the  data  was  standardized  using  the  SAS  procedure  STANDARD  and 
retested  using  the  maximum-likelihood  procedure  as  above. 
Appendix  B  provides  the  SAS  commands  used  for  the  exploratory 
factor  analysis. 

b.  Confirmatory  Factor  Analysis 

After  obtaining  the  optimal  factor  structure 
solution  via  exploratory  factor  analysis,  confirmatory  factor 
analysis  was  conducted.  This  time  multivariate  normality 
was  not  assumed  and  the  observed  variables  were  analyzed  for 
goodness  of  fit  to  the  optimal  exploratory  factor  analysis 
model  using  Joreskog  and  Sorbom's  LISREL  7  (Linear  Structural 
Relations)  computer  program.  The  LISREL  model  can  be  viewed 
in  terms  of  a  confirmatory  factor  analytic  model  [32]. 
Joreskog  and  Sorbom  [39]  in  their  LISREL  7  manual  provide 
testing  cases  for  non-normality  where  the  observed  variables 
are  on  interval  scales  using  Weighted  Least  Squares  analysis. 
Using  Joreskog  and  Sorbom's  [40]  PRELIS  program,  the  raw  data 
is  converted  and  saved  as  a  polychoric  correlation  matrix  and 
an  asymptotic  covariance  matrix  to  be  used  in  the  confirmatory 
factor  analysis  [39] [40].  Appendix  C  provides  the  PRELIS 
commands  used  and  the  LISREL  commands  used  for  confirmatory 
factor  analysis.  In  addition  to  the  Total  Coefficient  of 
Determination  (TCD) ,  Goodness  of  Fit  Index  (GFI) ,  Adjusted 
Goodness  of  Fit  Index  (AGFI) ,  and  Root  Mean  Square  Residual 
(RMR) ,  the  output  will  provide  the  Standard  Errors  (SE) ,  and 
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t-values  (TV).  LISREL's  t-values  or  critical  ratios  when 
larger  than  two  are  normally  judged  to  be  significantly 
different  from  zero,  and  therefore  indicating  a  true 
parameter  for  loading  in  that  factor  [39].  The  TCD  is  a 
measure  of  how  well  the  variables  jointly  serve  as 
measurement  instruments  for  the  overall  factor  structure.  The 
closer  to  1.0  the  TCD,  GFI,  and  AGFI  are,  the  better  the  model 
fits  the  data.  The  RMR  and  SE's  should  all  be  very  small  to 
indicate  overall  good  fit  of  the  data.  There  is  some  debate 
about  the  use  of  the  GFI.  Mulaik  et  al.  [35],  in  a  recent 
evaluation  recommends  the  use  of  GFI  when  one  has  samples  at 
least  200  in  size,  and  of  course  when  the  conditions  for  that 
method  are  satisfied. 

3.  Reliability  (Internal  Consistency)  Testing 

Internal  consistency  was  tested  for  each  factor  using 
Cronbach's  Alpha  reliability  coefficient  option  in  SAS's 
correlation  procedure  [41].  Cronbach's  Alpha  is  based  on  the 
average  correlation  of  items  within  a  factor  (or  test) .  It 
represents  the  correlation  between  this  factor  (or  test)  and 
all  other  possible  factors  (or  tests)  containing  the  same 
number  of  items,  which  could  be  constructed  from  a 
hypothetical  universe  of  items  that  measure  the 
characteristic  of  interest  (i.e.,  the  factor).  It  also  sets 
an  upper  limit  to  the  reliability  of  the  factor.  If  it 
proves  to  be  very  low,  either  the  factor  has  too  few  items  or 
the  items  have  very  little  in  common  [27].  According  to 
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Nunnally  [27],  reliability  coefficients  of  .70  or  higher  will 
suffice  in  the  early  stages  of  research,  and  for  basic 
research,  efforts  to  increase  much  beyond  .80  is  often  a 
waste  of  time  and  funds.  Appendix  D  provides  the  SAS 
commands  used  to  obtain  the  Cronbach  Alpha’s  for  each  factor. 

4.  Testing  for  Measurement  and  structure  Invariance 

Having  conducted  common  factor  analysis  on  the  data  as 
a  whole,  the  next  verification  is  to  test  that  the  factor 
structure  and  loading  are  the  same  for  each  of  the  three 
subpopulations.  In  addition,  before  conducting  ANOVA 
testing  of  the  three  subpopulation  location's  for  evaluating 
the  difference  in  means,  it  is  important  to  confirm  that  the 
measurement  and  the  structure  of  the  instrument  designed  to 
measure  attitudes  are  equivalent  across  the  subpopulations 
[37] [38] [42] .  Joreskog  and  Sdrbom  [39]  provide  a  methodology 
to  analyze  data  from  samples  simultaneously  using  their 
LISREL  models.  They  outline  a  series  of  tests  to  be 
conducted  to  confirm  measurement  and  structural  invariance. 
The  first  test  (hypothesis  A)  is  an  overall  test  of  the 
equality  of  covariance  matrices  (or  structures)  across  the 
three  subpopulations.  Failure  to  reject  the  null  hypothesis 
(i.e.,  covariance  matrices  are  equal)  is  statistical  evidence 
that  the  groups  can  be  treated  as  one. 

The  next  series  of  testing  consists  of  a  model  in 
which  certain  parameters  are  constrained  to  be  equal  across 
the  subpopulations  is  compared  with  a  less  restrictive  model 
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where  these  same  parameters  are  free  to  take  on  any  value 
[37] [39].  In  each  of  these  tests,  at  least  one  of  the  scales 
or  items  making  up  each  of  the  factors  must  be  fixed  to  1.0. 
In  this  study,  the  highest  loading  item  (from  previous  factor 
analysis  above)  was  fixed  to  1.0.  There  is  no  guidance  in 
the  literature  as  to  which  item  to  fix,  and  to  iteratively 
fix  the  various  items  in  each  factor  is  problematic.  The 
choice  uses  that  item  which  strongly  loads  into  its 
respective  factor,  and  therefore  setting  it  to  1.0  seemed 
appropriate.  The  next  test  (hypothesis  B)  tests  that  there 
are  four  factors  in  all  three  subpopulations  with  a  factor 
pattern  of;  Factor  A  consists  of  questions  1,  6,  11;  Factor 
B  consists  of  questions  2,  12;  Factor  C  consists  of  questions 
7,  8,  9,  10,  13;  and  Factor  D  consists  of  questions  3,  4,  and 
5.  Assuming  failure  to  reject  hypothesis  B,  hypothesis  C 
tests  that  there  is  invariance  in  factor  loadings  (lambda  x) 
across  the  subpopulations.  Assuming  failure  to  reject 
hypothesis  C,  hypothesis  D  tests  that  there  is  invariance  in 
the  error/uniqueness  (theta)  across  the  subpopulations. 
Lastly,  assuming  failure  to  reject  hypothesis  D,  hypothesis 
E  tests  that  the  factor  variances  and  covariances  (phi)  are 
invariant  across  the  three  subpopulations. 

For  hypothesis  B,  subpopulations  2  and  3  are  specified 
to  have  the  same  pattern  and  the  same  starting  values  as 
subpopulation  1  (LX=PS  command  on  the  LISREL  7  model  input 
line) .  In  hypothesis  C,  subpopulations  2  and  3  are  specified 
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to  be  invariant  for  factor  loading  from  subpopulation  1 
(LX=IN  command  on  the  LISREL  7  model  input  line) .  Hypothesis 
D  additionally  constrains  the  theta  matrices  to  be  invariant 
(TD=IN  command  on  the  LISREL  7  model  input  line) .  Hypothesis 
E  additionally  constrains  the  phi  matrices  to  be  invariant 
( PH=IN  command  on  the  LISREL  7  model  input  line) .  Appendix 
E  provides  the  PRELIS  and  LISREL  commands  used  for  each  of 
the  hypothesis  testing. 

The  LISREL  7  computer  program  output  provides  the  GFI 
and  RMR  for  each  subpopulation.  The  chi-square  measure 
provided  with  the  last  subpopulation  is  the  measure  of  the 
overall  fit  of  the  three  subpopulations.  Alternative  indices 
used  to  help  evaluate  LISREL  models  in  multiple  sample 
analysis  where  the  chi-square  measure  and  degrees  of  freedom 
are  reported  as  summed  values  from  the  multi-sample  testing 
(as  in  this  testing)  are:  the  chi-square  to  the  degrees  of 
freedom  ratio,  and  the  chi-square  likelihood  ratio  tests. 
The  chi-square/degrees  of  freedom  ratio  is  distributed  as  a 
t-statistic  so  that  anything  greater  than  1.96  (in  this 
instance,  where  n  =  340,  and  alpha  =  0.05)  is  significant 
[38].  The  chi-square  likelihood  ratio  (LR) ,  also  described  as 
the  chi-square  difference  test,  is  utilized  where  restricted 
nested  models  are  used  as  in  this  case  where  hypotheses  C,  D, 
and  E  are  restricted  nested  models  of  hypothesis  B.  The  LR 
test  is  calculated  by  taking  the  difference  in  the  chi-square 
estimators  for  the  restricted  and  unrestricted  models  and  the 
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difference  in  degrees  of  freedom  (df)  for  the  two  models  and 
reporting  as  a  chi-square/df  ratio  [37] [38] [43] . 

5.  Post  Survey  Interviews  (Convergent  Validity) 

Convergent  validity  is  the  extent  that  a  measure  is 
correlated  or  "agrees"  with  other  measures  of  the  same 
construct  [3].  Interviews  were  conducted  with  randomly 
selected  members  of  the  user  groups  at  the  Naval  Hospitals 
Camp  Lejeune,  and  Jacksonville.  The  interviews  were 
conducted  to  assess  users  overall  satisfaction  with  the 
system  for  comparison  with  the  instrument's  results. 
Additionally,  the  interviews  were  used  to  gain  comments  about 
the  system,  and  the  short-form  UIS  instrument  used  in  this 
study.  Subjects  interviewed  were  assured  that  their 
responses  would  be  kept  confidential. 

E.  EVALUATION  OF  THE  INSTRUMENT'S  DATA 

1.  Testing  Differences  in  Means 

Oneway  analysis-of-variance  (ANOVA)  testing  was 
conducted  as  described  above  for  testing  homogeneity  of 
sample  data  sets.  First,  the  combined  three  hospital  data 
set  was  used  to  test  the  difference  in  means  between  the 
three  user  groups  (physicians,  medical  pport,  and 
administrative  support)  for  each  of  the  13  questions,  overall 
score,  and  factors.  Using  the  combined  data  set,  each  user 
group  was  tested  for  differences  in  means  between  the  three 
location  sites  (i.e.,  Charleston  vs  Camp  Lejeune  vs 
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Jacksonville).  Lastly,  each  hospital's  data  was  tested  for 
differences  in  means  between  the  three  user  groups. 

Where  significant  differences  in  means  occurred  (i.e., 
P  value  less  than  0.05),  the  Scheffe  multiple  comparison 
testing  was  conducted  a  posteriori.  Additionally,  a 
posteriori  testing  for  normality  and  homogeneity  of  variance 
was  conducted.  Normality  was  tested  using  MINITAB 's  option 
for  computing  and  storing  fitted  and  residual  values  when 
performing  oneway  ANOVA  procedure.  Applying  the  NSCORE 
function  to  compute  the  normal  scores  of  the  residuals  and 
then  computing  the  correlation  of  the  normal  score  with 
residuals  approximates  a  normal  distribution  if  the 
correlation  is  large  (i.e.,  the  closer  to  1.0  the  better) 
[30].  The  Hartley's  Fmax  test  was  used  for  homogeneity  of 
variance. 

Nonparametric  Kruskal-Wallis  ANOVA  testing  of  sample 
medians  was  conducted  in  those  situations  where  parametric 
ANOVA  testing  assumptions  were  violated  (i.e.,  lack  of 
normality  and/or  homogeneity  of  variance) .  Using  a  PC 
version  of  MINITAB  statistical  program  [30],  the 
Kruskal-Wallis  test  statistic  H  and  P  values  were  calculated 
and  adjusted  for  ties  in  responses.  Where  significant 
differences  in  medians  occurred,  the  MINITAB  [30] 
nonparametric  Mann-Whitney  two-sample  median  procedure  was 
performed  in  pairwise  comparisons  to  identify  individual 
significant  differences. 
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2.  Time  of  Use  correlation  Testing 

The  assumption  held  is  that  as  the  length  of  time  of 
use  of  the  system  increased,  the  user's  level  of  satisfaction 
would  increase.  The  CHCS  is  a  mandatory  use  system  as 
opposed  to  an  optional  use  system.  As  such,  medical 
personnel  must  use  the  CHCS  to  accomplish  their  work  (in 
those  areas  where  respective  CHCS  modules  have  been 
installed) . 

Correlation  measures  the  degree  of  association  between 
two  variables.  The  range  of  correlation  strength  can  be  from 
-1.0  (perfect  negative  correlation)  to  +1.0  (perfect  positive 
correlation) ,  with  zero  meaning  no  correlation.  The  term 
"negative"  used  here  with  correlation  denotes  that  as  one 
variable  increases  the  other  variable  decreases.  The  term 
"positive"  used  with  correlation  denotes  that  as  one  variable 
increases  the  other  variable  increases.  The  independent 
variable  used  in  this  study  was  time  of  use  of  the  CHCS  (in 
terms  of  months) .  The  dependent  variable  to  test  against 
were  each  factor's  mean  scores.  The  Pearson's  sample 
correlation  coefficient  (r)  was  obtained  for  each  comparison 
using  MINITAB.  It  should  be  noted  that  rho  and  its  estimate 
r  are  both  symmetric  so  that  the  two  variables  to  be 
correlated  can  be  interchanged  without  changing  the  value. 
It  is  because  of  this  symmetry  that  no  cause  and  effect 
statement  may  be  made,  rather  just  the  strength  of 
association  or  relationship  between  the  two  variables  [31], 


The  Pearson's  sample  correlation  coefficient  was  tested  for 
significance  using  a  macro  program  provided  by  Zehna  [31]  for 
MINITAB.  The  testing  of  the  sample  correlation  coefficient 
(r)  used  one-tail  hypothesis  testing  where  HI:  rho  >  0  if  r 
was  positive  or  HI:  rho  <  0  if  r  was  negative  to  obtain  the 
appropriate  P  value.  A  P  value  of  less  than  0.05  indicates 
that  the  null  hypothesis  of  no  correlation  may  be  rejected. 

Correlations  were  performed  on  the  combined  three  hospital 
data. 

a.  Trend  Analysis 

Hurd  [1]  found  that  none  of  the  work  groups 
demonstrated  any  high  correlation  between  time  of  use  and  the 
level  of  satisfaction.  However,  he  used  six  month  time  series 
intervals  to  look  for  possible  trends  (negative  or  positive) 
between  the  time  of  use  of  the  CHCS  and  the  overall  UIS  summed 
score.  He  found  at  the  Naval  Hospital  Charleston,  that 
physicians  and  administrative  support  tended  to  exhibit  a 
positive  trend-line,  whereas,  the  ancillary  group  (which  is 
part  of  the  Medical  Support  group  in  this  study)  tended  to 
exhibit  a  negative  trend-line  for  overall  satisfaction. 

In  this  study,  the  trend  analysis  performed  by 
Hurd  [1]  was  replicated.  Trend  analysis  of  the  mean  score  in 
six  month  intervals  for  each  factor  was  conducted.  The 
number  of  individuals  involved  in  each  six  month  interval  and 
the  percentage  of  the  whole  were  tabulated  to  provide  clarity 
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as  to  the  weighing  of  the  results.  The  trend  analysis  was 
performed  on  the  combined  three  hospital  data. 
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IV.  DESCRIPTIVE  FINDINGS 


A.  DEMOGRAPHIC  FINDINGS  BY  LOCATION  AND  AS  A  GROUP 

Of  the  two  Naval  hospitals  surveyed  in  this  study,  121 
usable  instruments  were  obtained  from  the  Naval  Hospital  Camp 
Lejeune,  and  118  usable  instruments  were  obtained  from  the 
Naval  Hospital  Jacksonville.  The  response  rates  for  the 
Naval  Hospitals  Camp  Lejeune  and  Jacksonville  were  48%  and 
47%,  respectively.  These  rates  of  response  were  similar  to 
that  obtained  by  Hurd  [1]  in  his  study  where  he  had  101 
respondents  and  a  response  rate  of  56%.  Hurd's  data  from  the 
Naval  Hospital  Charleston  is  used  in  this  study.  Appendix  F 
contains  a  summary  of  the  demographics  by  location  and  as  a 
group  (the  three  hospitals  combined) . 

1.  Age 

Hurd's  [1]  data  revealed  an  average  age  of  respondents 
from  the  Naval  Hospital  Charleston  to  be  32  years,  with  a 
range  in  years  from  19  to  56.  The  average  age  of  respondents 
from  the  Naval  Hospital  Camp  Lejeune  was  33  years,  with  a 
range  in  years  from  18  to  61.  The  average  age  of  respondents 
from  the  Naval  Hospital  Jacksonville  was  32  years,  with  a 
range  in  years  from  19  to  56.  The  combined  group  had  an 
average  age  of  32  years  with  a  range  in  years  from  18  to  61. 
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2 .  Gender 


The  gender  of  the  respondents  from  the  Naval  Hospital 
Charleston  were  57  (56%)  male  and  44  (44%)  female.  The  gender 
of  the  respondents  from  the  Naval  Hospital  Camp  Lejeune  were 
73  (60%)  male  and  48  (40%)  female.  The  gender  of  the 
respondents  from  the  Naval  Hospital  Jacksonville  were  83 
(70%)  male  and  35  (30%)  female.  The  gender  split  in  the 
combined  group  of  respondents  were  213  (63%)  male  and  127 
(37%)  female. 

3 .  Hospital  Departments 

Only  outpatient  departments  were  sampled.  Work  areas 
reported  were  in  one  of  the  following  department  types: 
clinic,  administration,  laboratory,  pharmacy,  or  radiology. 
Figure  4.1  depicts  the  individual  and  combined  hospital 
department  types  and  percentages. 
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DEPARTMENT  WORK  CENTERS 


Figure  4.1.  Department  Work  Centers 


4 .  Job  Descriptions 

Job  descriptions  reported  were  categorized  into  one  of 
the  following  types: 

•  Physician  -  a  Medical  Corps  Officer  or  civilian 
equivalent. 

•  Nurse  -  a  Nurse  Corps  Officer  or  civilian  equivalent. 

•  Health  Professional  -  a  Medical  Service  Corps  Officer 
(Allied  Science)  or  civilian  equivalent. 

•  Administrator  -  a  Medical  Service  Corps  Officer 
(Health  Care  Administration)  or  civilian  equivalent. 

•  Technician  -  a  Hospital  Corpsman  with  a  medical 
technician  rating  or  civilian  equivalent. 

•  Corpsman  -  a  Hospital  Corpsman  without  a  technical 
rating. 

’  Clerical  -  a  person  performing  secretarial  or  clerical 
functions. 
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Figure  4 . 2  depicts  the  individual  and  combined 
hospital  job  description  types  and  percentages. 


JOB  DESCRIPTIONS 


Figure  4.2.  Job  Description 


5.  Functional  User  Groups 

Based  on  the  person's  work  department  and  job 
description  types  they  were  categorized  into  three  functional 
work  groups:  Physicians,  Medical  Support,  and  Administrative 
Support.  The  Physician  group  is  self  explanatory.  The 
Medical  Support  group  consisted  of:  nurses,  health 
professionals,  technicians,  and  corpsman  not  working  in  an 
administrative  department.  The  Administrative  Support  group 
consisted  of:  all  administrators,  and  clerical  persons 
regardless  of  department  assigned  to,  and  all  other  persons 


regardless  of  profession  assigned  to  an  administrative 
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department.  Figure  4.3  depicts  the  individual  and  combined 
hospital  user  group  types  and  percentages. 


USER  GROUPS 

(defined  by  member's  Work  Center  and  Job  Description) 


#12  3  4 


Figure  4.3.  User  Groups 


6.  Level  of  Education 

As  expected,  the  participants  working  within  a  health 
care  setting  possess  a  high  level  of  education.  Looking  at 
the  combined  hospital  data,  Figure  4.4,  almost  90%  of  the 
respondents  have  some  college  or  higher  educational 
experience.  The  median  education  level  of  the  respondents 
from  the  Naval  Hospital  Charleston  and  from  the  Naval 
Hospital  Camp  Lejeune  was  "some  college."  The  median 
education  level  of  the  respondents  from  the  Naval  Hospital 
Jacksonville  was  "bachelor  degree."  The  median  education 
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level  of  the  combined  group  was  "some  college."  Figure  4.4 
depicts  the  individual  and  combined  hospital  level  of 
education  and  percentages. 


Figure  4.4.  Education  Level 


7 .  Computer  Use 

Computer  use  is  not  an  unfamiliar  task  with  this  group 
of  participants.  80%  of  all  the  participants  have  used  other 
computer  systems  of  some  sort.  Of  these  users,  nearly  three 
quarters  of  them  (74%)  have  used  other  health  care  information 
system (s)  .  When  comparing  all  respondents,  59%  have  used 
other  health  care  information  system  (HCIS) .  The  respondents 
from  the  Naval  Hospital  Charleston  rf ported  a  80%  use  of 
other  computer  systems;  of  those  users,  71%  have  used  a  HCIS 
and  of  all  respondents,  58%  have  used  a  HCIS.  The 
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respondents  from  the  Naval  Hospital  Camp  Lejeune  reported  a 
76%  use  of  other  computer  systems;  of  those  users,  68%  have 
used  a  HCIS  and  of  all  respondents,  52%  have  used  a  HCIS. 
The  respondents  from  the  Naval  Hospital  Jacksonville  reported 
a  86%  use  of  other  computer  systems;  of  those  users,  76%  have 
used  a  HCIS  and  of  all  respondents,  66%  have  used  a  HCIS. 

8.  Length  of  Time  of  CHCS  Use 

Of  all  respondents,  the  average  length  of  time  in 
months  of  use  of  the  CHCS  was  12.3  months.  The  overall  time 
of  use  ranged  from  one  month  to  36  months.  The  respondents 
from  the  Naval  Hospital  Charleston  reported  an  average  time 
of  use  of  8.6  months,  with  a  range  of  one  to  19  months.  The 
respondents  from  the  Naval  Hospital  Camp  Lejeune  reported  an 
average  time  of  use  of  12.5  months,  with  a  range  of  one  to  36 
months.  The  respondents  from  the  Naval  Hospital  Jacksonville 
reported  an  average  time  of  use  of  15.3  months,  with  a  range 
of  one  to  32  months. 

B.  VALIDATION  OF  THE  INSTRUMENT 

1.  Homogeneity  of  the  Sample  Data  Sets 

The  oneway  ANOVA  testing  results  are  presented  in 
Table  4.1a.  Significant  differences  in  means  are  denoted  by 
underlining.  Significant  differences  in  subpopulation  means 
were  noted  in  the  education  and  the  time  of  use  of  CHCS  user 
attributes. 
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TABLE  4.1A 

USER  ATTRIBUTES  ANOVA  TESTING 


ATTRIBUTE 

F  W  .UE 

P  VALUE 

NSCORES 

CORRELATION 

HARTLEY 
t*AX(  1  I 

AGE 

0  16 

0  852 

0  98 

1 .59 

GEf*>Eft 

2  48 

0  086 

0.66 

1  18 

COMPUTER  SYSTE*  EXPERIENCE 

2  18 

0  115 

0.81 

1  55 

HOSPITAL  INFORftATfON  SYSTE*  EXPERIENCE 

2  45 

0  088 

0.89 

111 

EDUCATION 

4.9* 

0.018 

0.9J 

1.17 

Note 

(i)  Hartley  s  Fmax<}  n2)  approximate  critical  value  «i 

.803  alpha  *  0.05 

The  Scheffe  interval  comparison  testing  for  the 


education  attribute  is  presented  in  Table  4.1b.  Education 
was  tested  and  found  to  have  no  significant  correlation  with 
the  overall  satisfaction,  as  well  as  with  each  of  the  four 
factors  found  when  factor  analysis  was  performed.  Time  of 
use  of  the  CHCS  system  is  discussed  later  in  this  paper. 


TABLE  4. IB 

SCHEFFE  MULTIPLE  COMPARISON  TESTING  OF  EDUCATION  ATTRIBUTE 


SITE  LOCATION  COMPARISONS 

USER  ATTRIBUTE 

CHARLESTON/ 
CAMP  LEJEUNE 

CHARLESTON/ 

JACKSONVILLE 

CAMP  LEJEUNE/ 
JACKSONVILLE 

EDUCATION 

NS(1) 

NSC1) 

S(2) 

NOTES: 

<1)  NS  *  nonsignificant;  no  difference  between  the  means 
(2)  S  *  significant  difference  between  the  means 

2.  Exploratory  Factor  Analysis 

The  combined  data  set  (n=340)  was  used  to  perform  the 
exploratory  factor  analysis  procedure.  The  Cattell  Scree  Test 
suggested  a  maximum  of  five  factors  may  possibly  exist.  The 
SAS  maximum  likelihood  factor  analysis  procedure  was  written 
to  sequentially  perform  a  one-factor  solution  through  a 
five-factor  solution.  The  Kaiser’s  Measure  of  Sampling 
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Adequacy  (MSA)  had  a  value  of  0.89  indicating  a  very  good  fit 
of  the  data  for  factor  analysis.  The  maximum  likelihood 
factor  analysis  procedure  in  SAS  uses  squared  multiple 
correlations  (SMC)  as  its  initial  starting  estimate.  The  SMC 
is  the  lower  bound  for  the  reliability  of  each  variable.  The 
SAS  squared  canonical  correlations  (SCC)  reported  for  each 
factor  is  the  lower  bound  of  reliability  for  that  factor  (from 
the  variables  that  make  up  that  factor)  [33].  The  one-factor 
solution  made  up  of  all  13  questions  had  a  SCC  value  of  0.90. 

The  Schwarz's  Bayesian  Criterion  (SBC)  value  kept 
decreasing  for  the  two-factor  and  the  three-factor  solutions. 
This  indicated  that  a  greater  than  three-factor  solution  was 
optimal.  The  three-factor  solution  had  a  SBC  of  219  and  a 
Tucker  and  Lewis's  Reliability  Coefficient  of  0.91.  At  the 
four-factor  solution,  the  SBC  reached  its  lowest  value  of 
211,  and  rose  to  the  value  of  217  at  the  five-factor 
solution.  Table  4.2  shows  the  optimal  four-factor  solution. 
The  Tucker  and  Lewis's  Reliability  Coefficient  was  0.95  for 
the  four-factor  solution. 

Factor  A  is  made  up  of  questions  1,  6,  and  11,  and 
represents  the  local  Management  Information  Department  (MID) 
staff  and  services.  Factor  B  is  made  up  of  questions  2,  and 
12,  and  represents  the  contractor's  services.  Factor  C  is 
made  up  of  questions  7,  8,  9,  10,  and  13,  and  represents  the 
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information  product  output.  Factor  D  is  made  up  of  questions 
3,  4,  and  5,  and  represents  user  knowledge  and  involvement. 

3.  Confirmatory  Factor  Analysis 

Paramount  to  the  maximum  likelihood  factor  analysis 
procedure  is  that  the  data  be  multivariate  normal.  There  is 
no  easily  defined  test  for  multivariate  normality  that  could 
be  found  in  the  literature.  Instead,  the  Joreskog  and 
Sorbom's  [39]  LISREL  model  for  analysis  of  non-normal 
variables  was  used.  The  Total  Coefficient  of  Determination 
(TCD)  for  the  variables  was  0.997  indicating  a  very  good  fit 
to  the  four-factor  solution.  Other  goodness  of  fit  indices 
used  supported  the  four-factor  solution.  The  Goodness  of  Fit 
Index  (GFI)  was  0.985,  and  the  Adjusted  Goodness  of  Fit  Index 
(AGFI )  was  0.977. 

The  Root  Mean  Square  Residual  (RMR)  was  0.041,  which 
also  supports  the  goodness  of  fit  of  the  four-factor 
solution.  Additionally,  Standard  Errors  (SE)  and  t-values 
(LISREL1 s  critical  ratios)  were  performed  for  each  of 
variable's  loading  into  their  respective  factor.  The  SE's 
were  all  low  (<.04),  and  the  t-values  were  all  large  (>20) 
for  each  of  variable  factor  loadings,  which  further  supports 
the  goodness  of  fit  of  the  four-factor  solution. 

4.  Reliability  (Internal  Consistency)  Testing 

Factor  A  is  made  up  of  questions  1,  6,  and  11,  and 
represents  the  local  Management  Information  Department  (MID) 
staff  and  services.  Factor  B  is  made  up  of  questions  2,  and 
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12,  and  represents  the  contractor's  services.  Factor  C  is 
made  up  of  questions  7,  8,  9,  10,  and  13,  and  represents  the 
information  product.  Factor  D  is  made  up  of  questions  3,  4, 
and  5,  and  represents  knowledge  and  involvement. 

The  Cronbach's  alpha  for  Factor  A  was  0.89;  for  Factor 
B  was  0.68;  for  Factor  C  was  0.87;  and  for  Factor  D  was  0.75. 


TABLE  4.2 

MAXIMUM  LIKELIHOOD  FACTOR  ANALYSIS  WITH  VARIMAX  ROTATION 


QUEST  1 0NS/VAR 1 A8L  ES 

FACTOR  A 

MID  STAFF  AISD 
SERVICES 

factor  b 

CONTRACTOR 

SERVICES 

FACTOR  C 
INFORMATION 
OUTPUT 

FACTOR  D 
KNOWLEDGE  ASD 
INVOLVEMENT 

SMC 

J . 

RELATIONSHIP  WITH  MID  STAFF 

0.75 

0.18 

0  16 

0.20 

0.61 

2. 

PROCESSING  OF  REQUESTS  FOR 
CHANGES 

0  12 

0 . 51 

0.  19 

0  16 

0.36 

3. 

DEGREE  OF  TRAINING  PROVIDED 

0.27 

0.34 

0  16 

0-41 

0.43 

4  . 

USER  S  UNDERSTATING  OF 

SYSTEM 

0  16 

0.11 

0  10 

0.79 

0  37 

5. 

USERS  FEELING  OF 

PARTICIPATION 

0  25 

0.32 

0.30 

0.51 

0.48 

6 

ATTITUDES  OF  MlO  STAFF 

0.79 

0  14 

0  20 

0  15 

0.63 

7. 

RELIABILITY  OF  OUTPUT 

0  16 

0.25 

0  77 

0.07 

0.61 

8. 

RELEVANCY  OF  OUTPUT 

0  23 

0.32 

0-61 

0  15 

0.54 

9 

ACCURACY  OF  OUTPUT 

0  20 

0  15 

0.75 

0  09 

0  56 

10 

PRECISION  OF  OUTPUT 

0  12 

0.05 

0-71 

0  19 

0  48 

11 

COMMUNICATING  WITH  MID  STAFF 

0->3 

0.14 

0.22 

0  19 

0.69 

12. 

TIME  REQUIRED  FOR  NEW 

DEVELOPMENT 

0  17 

0.69 

0.23 

0.  19 

0.45 

13 

COMPLETENESS  OF  OUTPUT 

0  20 

0.44 

0.59 

0  10 

0  57 

E i genva 1 uc 

4  10 

2  30 

15  90 

1  30 

-- 

Cumuiat ive  percent 

17% 

26 % 

94% 

100% 

- 

see 

0  to 

0  70 

0  94 

0  56 

— 

Cronbach's  Alpha 

0  69 

0  68 

0.87 

0.75 

- 

Schwarz  Bayesian  criterion  *  211 

Tucker  and  Lewis  Reliability  coefficient  «  0  95 


5.  Testing  for  Measurement  and  structural  Invariance 

The  use  of  Joreskog  and  Sorbom's  LISREL  model  to  test 
for  measurement  and  structural  invariance  revealed  that  the 
three  subpopulations  are  equivalent  for  their  responses  and 
the  four-factor  data  reduction.  Hypothesis  A  was  that  the 
covariance  structure  across  the  three  subpopulations  is 
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invariant.  Hypothesis  B  was  that  the  number  of  factors  of 
the  factor  structure  is  the  same  across  the  three 
subpopulations  is  invariant.  Hypothesis  C  was  that  the 
factor  loading  pattern  across  the  three  subpopulations  is 
invariant.  Hypothesis  D  was  that  the  error/uniqueness 
structure  is  invariant  across  the  three  subpopulations. 
Hypothesis  E  was  that  the  factor  variances  and  covariances 
are  invariant  across  the  three  subpopulations. 

As  mentioned  previously  in  Chapter  III,  the 
chi-square/degrees  of  freedom  ratio  is  distributed  as  a 
t-statistic  so  that  anything  greater  than  1.96  is  significant 
and  therefore  would  reject  the  null  hypothesis  of  invariance. 
Table  4.3  shows  the  testing  results. 

TABLE  4.3 

EQUIVALENCE  TESTING 

HYPOTHESIS  cm2 _ df _ chi2/dt _ LR  Test 

A:  COVARIANCE  STRUCTURE  IS  THE  SAME  242.4  182  I  33 

B  NUMBER  Of  FACTORS  IS  THE  SAME  331.2  177  1.87 

C  FACTOR  LOADING  IS  THE  SAME  34 1  0  195  I  75  0  55 

D  THE  ERROR/UNIQUENESS  IS  THE  SAME  383  6  221  1  74  1  19 

E  fACTOR  VARIANCES  A  M3  COVARIANCES  ARE  THE  SAME _ 394  5 _ 241 _ 1  64  0  99 

6.  Post  Survey  Interviews  (Convergent  Validity) 

During  the  one-day  visits  to  the  Naval  Hospitals  Camp 
Lejeune  and  Jacksonville,  time  constraints  limited  the  number 
of  individuals  that  were  interviewed  to  a  total  of  15 
(approximately  5%  of  the  total  sample  population) .  In 
working  around  individual  work  schedules,  an  equal 


distribution  of  interviews  was  obtained.  Of  the  total  15 
interviews,  three  were  with  physicians,  three  with 
administrative  personnel,  three  with  clinic  personnel,  three 
with  laboratory  personnel,  and  three  with  pharmacy  personnel. 
During  the  interviews,  the  individuals  were  asked  about  their 
overall  impression  of  satisfaction  (satisfied  or 
dissatisfied) ,  and  to  comment  about  the  system.  All 
individuals  interviewed  had  at  least  nine-months  experience 
with  the  CHCS .  Between  interviews,  there  was  an  opportunity 
for  first-hand  use  of  the  CHCS  at  the  Naval  Hospital  Camp 
Lejeune's  training  room. 

The  majority  of  the  physicians  (2  out  of  3)  were 
overall  dissatisfied  with  the  CHCS.  They  sited  a  cumbersome 
menu  interface,  slow  response  time  of  system,  and  slow 
response  time  to  change  the  system.  The  clinic, 
administrative,  laboratory  and  pharmacy  personnel  interviewed 
expressed  they  were  satisfied,  but  also  echoed  the  same 
comments  as  the  physicians. 

Other  general  comments  offered  about  the  survey 
instrument  were:  1)  provide  an  example — the  dual  bipolar 
adjective  pairs  tended  to  confuse  some  and  2)  the  instrument 
did  not  address  how  the  user  interacted  or  put  information 
into  the  system. 
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C.  EVALUATION  OF  THE  INSTRUMENT'S  DATA 


1.  Testing  Differences  in  Means 

The  combined  three  hospital  data  set  was  used  to  test 
the  difference  in  means  between  the  three  user  groups 
(Physicians,  Medical  Support,  and  Administrative  Support)  for 
each  of  the  13  questions,  overall  (summed)  score,  and  factors 
(averaged  score) .  Using  the  combined  data  set,  each  user 
group  was  tested  for  differences  in  means  between  the  three 
location  sites  (i.e.,  Charleston  vs  Camp  Lejeune  vs 
Jacksonville).  Lastly,  each  hospital's  data  was  tested  for 
differences  in  means  between  the  three  user  groups.  Figure 
4.5  shows  the  level  of  satisfaction  for  each  of  the  13 
questions  and  by  each  user  group. 

MEASURE  OF  USER  SATISFACTION 


BY  INDIVIDUAL  QUESTIONS  AND  USER  GROUPS 


Figure  4.5.  Measure  of  User  Satisfaction 
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a.  Combined  Hospital  Data 

(1)  Differences  between  User  Groups 

Table  4.4a  shows  the  ANOVA  testing  results 
between  user  groups.  Significant  findings  (at  an  alpha  of 
0.05)  are  underlined.  The  Physician  group  sample  size 
consisted  of  79  participants;  the  Medical  Support  group 
consisted  of  207  participants;  and  the  Administrative  Support 
group  consisted  of  54  participants.  The  a  posteriori 
testing  for  normality  via  the  NSCORES  correlation 
demonstrates  that  the  data  has  a  normal  distribution.  Except 
for  questions  9  and  10,  the  Hartley  Froax  test  revealed 
homogeneity  of  variance  between  groups.  Questions  9  and  10 
were  tested  using  the  Kruskal-Wallis  nonparametric  ANOVA 
testing  of  sample  medians.  The  nonparametric  findings  also 
showed  no  significant  difference  in  medians.  Questions  7,  8, 
9,  10,  and  13  which  make  up  Factor  C  (information  product 
output)  revealed  no  significant  differences.  Factor  C  across 
the  three  user  groups  means  were  within  the  "slightly 
satisfied"  range  (0  to  1) . 
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TABLE  4.4A 

COMBINED  HOSPITALS  ANOVA  TESTING;  USER  GROUPS 


QUESTIONS/FACTORS 

USER 

GROUP  MEANS! 1 ) 

F  VALUE 

P  VALUE 

NSCORES 

Hartley 

Fmax(2) 

P 

M 

A 

1.  RELATIONSHIP  WITH  MID  STAfF 

0.88 

1.32 

1.65 

6.22 

0.002 

0.98 

1.12 

2.  PROCESSING  Of  REQUESTS  FOR  CHANGE 

-0.83 

-0.28 

-0.24 

4.02 

0.019 

0.98 

1.54 

3.  DEGREE  OF  TRAINING 

-0.47 

0.39 

0.12 

7.23 

0.001 

0.99 

1.34 

4  USER  S  UNDERSTAND  1 NG  OF  SYSTEM 

-0.19 

0.98 

1.03 

<**> 

0.000 

0.98 

1.17 

5  USERS  FEELING  OF  PARTICIPATION 

-0.11 

0.94 

0.81 

14.24 

0.000 

0.96 

1.17 

6.  ATTITUDE  OF  MID  STAFF 

1.12 

1 .45 

1.79 

4.37 

0.013 

0.96 

1.13 

7  RELIABILITY  OF  OUTPUT 

1.10 

0.98 

0.94 

0.29 

0.752 

0.97 

1.41 

8  RELEVANCY  OF  OUTPUT 

0.60 

1.04 

1.02 

2.94 

0.054 

0.97 

1.11 

9.  ACCURACY  OF  OUTPUT 

1.32 

1.20 

0.81 

2.43 

0.089 

0.97 

2.08 

10.  PRECISION  Of  OUTPUT 

0.94 

0.72 

0.60 

1.26 

0.285 

0.98 

2.01 

It.  COMMUNICATION  with  mid  STAFF 

0.79 

1  .  17 

1.53 

4. W9 

0.007 

0.96 

1.35 

12.  TIME  REQUIRED  FOR  NEW  DEVELOPMENT 

-1.04 

0.05 

-0.15 

13.QS 

0.000 

0.99 

1.26 

13.  COMPLETENESS  OF  OUTPUT 

0.45 

0.77 

0.73 

1.28 

0.279 

0.98 

1 . 16 

OVERALL  SCORE 

4.54 

10.71 

10.62 

7.31 

0.001 

0.99 

1.69 

A.  MID  STAFF  AID  SERVICES 

0.93 

1.31 

1.66 

6.49 

0.002 

0  98 

1.13 

B  CONTRACTOR  SERVICES 

-0.95 

-0.12 

-0.19 

10.45 

0.000 

0.99 

1.35 

C.  INFORMATION  OUTPUT 

0.88 

0.94 

0.82 

0.26 

0.772 

0.96 

1.61 

D  KNOWLEDGE  AID  INVOLVEMENT 

-0.26 

0.77 

0.65 

10.70 

0.000 

0.99 

1.28 

(?ICSP  «  Physicians;  m  *  Medical  support,  a  «  Administrative  support 
(2)  Hartley's  Fmax(3  112)  approximate  critical  value  *  1.80  at  alpha  ■ 

0.05 

(2)  Significant  User  Group  Findings 

Table  4.4b  represents  those  items  where  the 
ANOVA  testing  in  Table  4.4a  revealed  a  significant  difference 
in  the  means  between  the  user  groups.  Scheffe  multiple 
comparison  testing  was  used  to  identify  the  individual 
differences  between  user  groups.  Physicians  were  less 
satisfied  and  displayed  a  significant  difference  between  the 
other  two  user  groups  in  virtually  all  the  individual 
questions  that  make  up  Factor  A  (MID  Staff  and  Services; 
questions  1,  6,  and  11)  ,  Factor  B  (Contractor  Services; 

questions  2  and  12)  ,  and  Factor  D  (User  Knowledge  and 
Involvement;  questions  3,  4,  and  5). 


TABLE  4.4B 

SCHEFFE  MULTIPLE  COMPARISON  TESTING;  USER  GROUPS 


USER  GROUPS  COMPARISON 

QUEST  I0NS/F ACTORS 

PHYSICIAN/ 
MEDICAL  SUPPORT 

PHYSICIAN/ 

ADMIN  SUPPORT 

MEDICAL  SUPPORT/ 
AOMIN  SUPPORT 

1 . 

KELAT ionship  with  mid  staff 

S(t) 

S 

NS(  2) 

2. 

PROCESS  INC  OF  REQUESTS  FOR  CHANCES 

s 

NS 

NS 

3. 

DECREE  OF  TRAINING  PROVIDED 

s 

NS 

NS 

4. 

USER'S  UADERSTANDINC  OF  SYSTEM 

s 

S 

NS 

3. 

USERS  FEELING  OF  PARTICIPATION 

s 

s 

NS 

6. 

ATTITUOES  OF  MID  STAFF 

NS 

s 

NS 

II, 

.  COMMUNICATING  WITH  MID  STAFF 

NS 

s 

NS 

12 

.  TIME  REQUIRED  FOR  NEW  DEVELOPMENT 

s 

s 

NS 

OVERALL  SCORE 

s 

s 

NS 

A. 

MID  STAFF  AM)  SERVICES 

s 

s 

NS 

B 

CONTRACTOR  SERVICES 

s 

s 

NS 

C. 

KNOWLEDGE  AM)  INVOLVEMENT 

s 

s 

NS 

NOTES: 

<lj  S  ■significant  difference  In  means 
(2)  NS  ■  Nons l gnl cant ;  no  difference  in  means 

(3)  Physician  Group  Differences  by  site 

Table  4 . 5a  shows  the  ANOVA  testing  results 
for  Physician  user  group  between  the  three  Naval  hospital 
(NH)  sites.  Significant  findings  (at  an  alpha  of  0.05)  are 
underlined.  The  Physician  group  sample  size  at  the  NH 
Charleston  consisted  of  21  participants;  NH  Camp  Lejeune 
consisted  of  22  participants;  and  NH  Jacksonville  consisted 
of  36  participants.  The  a  posteriori  testing  for  normality 
via  the  NSCORES  correlation  demonstrates  that  the  data  has  a 
normal  distribution.  Except  for  questions  7  and  13,  the 
Hartley  Fmax  test  revealed  homogeneity  of  variance  between 
groups.  Questions  7  and  13  were  tested  using  the 
Kruska 1-Wall is  nonparametric  ANOVA  testing  of  sample  medians. 
The  nonparametric  findings  also  showed  no  significant 
difference  in  medians. 
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Questions  7,  8,  9,  10,  and  13  which  make  up 
Factor  C  (information  product  output)  revealed  no 
significant  differences.  Factor  C  across  the  three  sites  for 
the  Physician  group  were  essentially  within  the  '’slightly 
satisfied"  range  (0  to  1) .  Other  nonsignificant  differences 
in  means  were  found  for  questions  4  and  12.  Questions  4  and 
12  both  were  essentially  within  the  "slightly  dissatisfied" 
range  (-1  to  0) ,  and  help  make  up  Factors  D  and  B 
respectively . 
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TABLE  4.5A 

PHYSICIAN  GROUP  ANOVA  TESTING 


QUEST I0NS/F ACTORS 

LOCATION  SITE  MEANS(I) 

F  VALUE 

P  VALUE 

_  NSC0RES_ 

hartley 

CH 

CL 

JX 

I. 

RELATIONSHIP  WITH  *10  STAff 

138 

1  36 

0.29 

0.23 

0.000 

098 

1.70 

2 

PROCESSING  Of  REQUESTS  FOR  CHANGE 

-0.45 

-0.05 

-1  60 

0.01 

0.000 

0  98 

1  45 

3. 

DECREE  Of  TRAINING 

0.05 

0.05 

-1  08 

5.99 

0-004 

0.99 

1  48 

4. 

USER'S  UNDERSTANDING  Of  SYSTEM 

0.  14 

-0.59 

-0.  14 

1.26 

0.269 

0  96 

1.23 

5. 

USERS  FEELING  Of  PARTICIPATION 

0.55 

-0.21 

-0.44 

3.49 

0.035 

0.97 

1  .69 

6. 

ATTITUDE  Of  «(0  STAFF 

1.57 

1.77 

0.46 

12.06 

0.000 

0  97 

1.72 

7. 

RELIABILITY  OF  OUTPUT 

1.07 

1.32 

0.99 

0.47 

0.625 

0.95 

3.13 

a 

RELEVANCY  OF  OUTPUT 

0.57 

0.71 

0.54 

0.09 

0.913 

0.96 

1 . 10 

9. 

ACCURACY  OF  OUTPUT 

1.41 

1.50 

1  .  17 

0.80 

0.451 

0.97 

1  98 

10 

PRECISION  OF  OUTPUT 

0.86 

0.66 

1.04 

0.32 

0.728 

0.98 

1.76 

11 

COMMUNICATION  WITH  *10  STAFF 

1.36 

1.34 

0.  13 

13.61 

0-000 

0  99 

1.40 

12 

TIME  REQUIRED  FOR  NEW  DEVELOPMENT 

-0  93 

-0  52 

-1.42 

2.61 

0  075 

0  98 

1  15 

13 

COMPLETENESS  OF  OUTPUT 

0  52 

0  84 

0  17 

1.51 

0.227 

0  98 

5.07 

OVERALL  SCORE 

8  10 

6  39 

0  11 

7.21 

0.001 

0.97 

1  68 

A. 

MID  STAFF  AM)  SERVICES 

1  .44 

1  49 

0  29 

14.69 

0-001 

0.98 

1  24 

B 

CONTRACTOR  SERVICES 

-0  69 

-0.28 

-1.50 

7.34 

0.001 

0  99 

1.74 

C 

INFORMATION  OUTPUT 

0  86 

1  04 

0  78 

0.51 

0.603 

0  97 

2.  14 

0. 

KNOWLEDGE  AM>  INVOLVEMENT 

0  25 

-0.25 

-0.56 

3.23 

0.045 

0.99 

2  05 

NOTES: 

(1)  CH  *  m  charleston:  CL  •  M-f  Came 

(2)  Hartley's  F«ax(3  77)  approximate  cri 

jeyne:  jx  *  w 
it  leal  value  * 

/acKsonvi 1 le 
5.37  at  alpha 

«  0  05 

(4)  Significant  Physician  Group  Findings 

Table  4 . 5b  represents  those  items  where  the 
ANOVA  testing  in  Table  4.5a  revealed  a  significant  difference 
in  the  means  between  the  Physician  groups.  Scheffe  multiple 
comparison  testing  was  used  to  identify  the  individual 
differences  between  sites. 

Physicians  at  the  NH  Jacksonville  were  less 
satisfied  and  displayed  a  significant  difference  between  the 
other  two  sites  in  all  the  individual  questions  (1,  6,  and 
11)  that  make  up  Factor  A  (MID  Staff  and  Services) .  The  NH 
Charleston  and  the  NH  Camp  Lejeune  Factor  A  findings  were 
within  the  "quite  satisfied"  range  (1  to  2)  versus  the  NH 
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Jacksonville's  findings  within  the  ''slightly  satisfied"  range 
(0  to  1)  .  Physicians  at  the  NH  Jacksonville  were  less 
satisfied  and  displayed  a  significant  difference  between  the 
other  two  sites  for  Factor  B  (Contractor  Services;  questions 
2  and  12) .  Physicians  at  the  NH  Jacksonville  were  less 
satisfied  as  compared  with  the  NH  Charleston  with  Factor  D 
(Knowledge  and  Involvement;  questions  3,  4,  5),  but 
demonstrated  no  difference  between  the  NH  Camp  Lejeune.  The 
NH  Jacksonville,  and  the  NH  Camp  Lejeune  findings  for  Factor 
D  were  within  the  "slightly  dissatisfied"  range  (-1  to  0) 
verses  the  NH  Charleston's  findings  within  the  "slightly 
satisfied"  range  (0  to  1). 
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TABLE  4.5B 

SCHEFFE  MULTIPLE  COMPARISON  TESTING;  PHYSICIAN  GROUP 


LOCATION  SITE  COMPARISONS 

QUEST  I0NS/FACT0RS 

CHARLESTON/ 

CHARLESTON/ 

CAMP  LEJEUNE/ 

CAMP  LEJEUNE 

JACKSONVILLE 

t.  RELATIONSHIP  WITH  MID  STAFF 
2.  PROCESSING  Of  tequtSTS  FOR  CHANCES 

I.  DEGREE  Of  TRAININC  PROVIDED 

5.  USER'S  FEELING  Of  PARTICIPATION 

6.  ATTITUDES  Of  «IO  STAFF 

II.  COWRUNICATION  WITH  RIO  STAFF 
OVERALL  SCORE 

A.  RID  STAFF  AFC  SERVICES 

B.  CONTRACTOR  SERVICES 

0  KNOWLEDGE  AFC  INVOLVEMENT 


(5)  Medical  Support  Group  Differences  by  Site 

Table  4.6a  shows  the  ANOVA  testing  results 
for  Medical  Support  user  group  between  the  three  naval 
hospital  (NH)  sites.  Significant  findings  (at  an  alpha  of 
0.05)  are  underlined.  The  Medical  Support  group  sample  size 
at  the  NH  Charleston  consisted  of  71  participants;  at  the  NH 
Camp  Lejeune  consisted  of  75  participants;  and  at  the  NH 
Jacksonville  consisted  of  61  participants.  The  a  posteriori 
testing  for  normality  via  the  NSCORES  correlation  demonstrates 
that  the  data  has  a  normal  distribution.  The  Hartley  Fmax 
test  revealed  homogeneity  of  variance  between  groups. 

Questions  7,  8,  9,  10,  and  13  which  make  up 
Factor  C  (information  product  output)  revealed  no  significant 
differences.  Factor  C  across  the  three  user  groups  means  were 
essentially  within  the  "slightly  satisfied"  range  (0  to  1)  . 
Other  nonsignificant  differences  in  means  was  found  for 


56 


question  4.  Question  4  was  essentially  within  the  "slightly 
dissatisfied"  range  (0  to  -1) ,  and  helps  make  up  Factor  D. 


TABLE  4.6A 

MEDICAL  SUPPORT  GROUP  ANOVA  TESTING 


LOCATION  SITE  ME ANS( 

) 

F  VALUE 

P  VALUE 

NSCORES 

mood  iTinu 

Hart  le 

lal  I/O 

QUEST  IONS/ FACTORS 

CH 

CL 

)X 

UIKNCLAI  ION 

rNX<  l 

1.  RELATIONSHIP  WITH  MID  STAFF 

1.74 

1.22 

0.94 

6  18 

0.001 

0.96 

1.24 

2  PROCESSING  Of  REQUESTS  FOR  CHANGE 

-0  42 

0.24 

-0  76 

6.79 

0.001 

0.99 

1.69 

3  DEGREE  OF  TRAINING 

0.52 

0.54 

-0.30 

7.37 

0.001 

0.98 

1.05 

4  USER'S  UM)ERSTAM)ING  OF  SYSTEM 

1.33 

0.82 

0.77 

2.58 

0.078 

0.97 

1.42 

5.  USER'S  FEELING  OF  PARTICIPATION 

1.25 

099 

0.53 

3.66 

0.027 

0.98 

1.32 

6  ATTITUDE  OF  MID  STAFF 

2  08 

1.33 

0  85 

16.91 

0.000 

0.98 

1.98 

7  RELIABILITY  OF  OUTPUT 

0.80 

1.27 

0  81 

2.34 

0  099 

0.97 

1.48 

«  RELEVANCY  OF  OUTPUT 

1.06 

1.23 

0.76 

1.90 

0.151 

0.98 

1.26 

9  ACCURACY  OF  OUTPUT 

1.31 

1.33 

0.91 

1  .66 

0.  189 

0.97 

1.20 

10  PRECISION  OF  OUTPUT 

0  58 

0.79 

0.79 

0  53 

0.591 

0.97 

1.22 

11.  COMMUNICATION  WITH  MID  STAFF 

1  73 

1.03 

0  68 

10.92 

0.000 

0.98 

1  .64 

12.  TIME  REQUIRED  FOR  NEW  DEVELOPMENT 

0.  10 

0.34 

-0.36 

3-08 

0.046 

0.99 

132 

13  COMPLETENESS  Of  OUTPUT 

0  66 

0  94 

0.67 

0.63 

0  536 

0.98 

1.21 

OVERALL  SCORE 

13.06 

12  08 

6.30 

502 

0.007 

0  99 

1.23 

A.  MID  STAFF  AM)  SERVICES 

1.85 

1.20 

0.83 

14.10 

0.000 

0  <>8 

1  .64 

B.  CONTRACTOR  SERVICES 

-0  16 

0.29 

-0.56 

6.09 

0.003 

0  99 

1.47 

C  INFORMATION  OUTPUT 

0.89 

1.11 

0.79 

1.27 

0.264 

0  98 

1.26 

0.  KNOWLFDCE  AND  INVOLVEMENT 

1  13 

0,78 

0.33 

6.36 

0.002 

0  99 

1.29 

NOTES: 

(1)  CH  «  w  Charleston;  CL  »  Mi  Camp 

(2)  Hartley's  Fiaax(-«  approxlaate 

le 

cr 

eune: 
t  leal 

JX  ■  Ml 

value  * 

jacksonvi 1 le 
1.84  at  alpha  « 

0.05 

(6)  Significant  Medical  Support  Group  Findings 

Table  4 . 6b  represents  those  items  where  the 
ANOVA  testing  in  Table  4.6a  revealed  a  significant  difference 
in  the  means  between  the  Medical  Support  groups.  Scheffe 
multiple  comparison  testing  was  used  to  identify  the 
individual  differences  between  sites.  For  Factor  A  (MID 
Staff  and  Services)  comprised  of  questions  1,  6,  and  11,  the 
NH  Charleston  was  significantly  different  from  the  other  two 


sites.  In  each  of  the  individual  questions  the  NH  Charleston 
was  more  satisfied  ("quite  satisfied"  range  of  1  to  2)  than 
the  two  sites  which  were  within  the  "slightly  satisfied" 
range  (0  to  1) .  For  Factor  B  (Contractor  Services) ,  which 
was  comprised  of  questions  2  and  12,  the  NH  Jacksonville  was 
less  satisfied  when  compared  with  the  NH  Camp  Lejeune,  but  had 
no  significant  difference  between  the  NH  Charleston.  Both 
the  NH  Jacksonville  and  the  NH  Charleston  findings  for  Factor 
B  were  within  the  "slightly  dissatisfied"  range  (0  to  -1) , 
whereas  the  NH  Camp  Lejeune  findings  for  Factor  B  were  within 
the  "slightly  satisfied"  range  (0  to  1).  Even  though  the  NH 
Charleston  and  the  NH  Camp  Lejeune  have  opposing  findings  for 
Factor  B,  they  were  still  statistically  nonsignificant  for 
differences  between  their  respective  means.  For  Factor  D 
(User  Knowledge  and  Involvement) ,  the  NH  Jacksonville  scored 
less  satisfied  findings  as  compared  with  the  NH  Charleston, 
but  no  significant  difference  between  the  NH  Camp  Lejeune. 
However,  all  three  sites  for  Factor  D  scored  within 
essentially  the  "slightly  satisfied"  range  (0  to  1) . 
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TABLE  4.6B 

SCHEFFE  MULTIPLE  COMPARISON  TESTING;  MEDICAL  SUPPORT  GROUP 


LOCATION  SITE  COMPARISONS 

QUEST  JONS/f ACTORS 

CHARLESTON/ 
CAMP  LEJEUNE 

CHARLESTON/ 

JACKSONVILLE 

CAMP  LEJEUNE/ 
JACKSONVILLE 

1 . 

RELATIONSHIP  WITH  MID  STAFF 

set) 

S 

NS(  2) 

2 

PROCESSING  OF  REQUESTS  FOR  CHANGES 

NS 

NS 

S 

S 

DECREE  OF  TRAINING  PROVIDED 

NS 

S 

s 

5 

USER'S  FEELING  OF  PARTICIPATION 

NS 

s 

NS 

6. 

ATTITUDES  OF  MID  STAFF 

S 

s 

NS 

1 1 

COMMUN 1  CAT  INC  WITH  MID  STAFF 

s 

s 

NS 

12 

TIME  REQUIRED  FOR  NEW  DEVELOPMENT 

NS 

NS 

NS 

OVERALL  SCORE 

NS 

s 

S 

A. 

MID  STAFF  AND  SERVICES 

s 

s 

NS 

8 

CONTRACTOR  SERVICES 

NS 

NS 

S 

D 

KNOWLEDGE  AM)  INVOLVEMENT 

NS 

s 

NS 

*J0T|S 

(1)  s  «  Significant  difference  In  means 

(2)  ns  «  Nonsignificant:  no  difference  in  means 

(7)  Administrative  Group  Differences  by  Site 

Table  4.7a  shows  the  ANOVA  testing  results 
for  Administrative  Support  user  group  between  the  three  naval 
hospital  (NH)  sites.  Significant  findings  (at  an  alpha  of 
0.05)  are  underlined.  The  Administrative  Support  group 
sample  size  at  the  NH  Charleston  consisted  of  9  participants; 
NH  Camp  Lejeune  consisted  of  24  participants;  and  NH 
Jacksonville  consisted  of  21  participants.  The  a  posteriori 
testing  for  normality  via  the  NSCORES  correlation 
demonstrates  that  the  data  has  a  normal  distribution.  Except 
for  Factor  B,  the  Hartley  Fmax  test  revealed  homogeneity  of 
variance  between  groups. 

Factor  B  which  is  the  averaged  score  of 
questions  2  and  12,  was  tested  using  the  Kruskal-Wallis 
nonparametric  ANOVA  testing  of  sample  medians.  The 
nonparametric  findings  showed  significant  difference  in 
medians  (P  value  =  0.01).  However,  the  Mann-Whitney 
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nonparametric  pairwise  comparisons  were  the  same  as  the 
Scheffe  parametric  findings  as  shown  in  Table  4.7b. 

Factor  D  dealing  with  training,  knowledge  and 
involvement  with  the  CHCS  system  was  nonsignificant  for 
differences  in  means  between  the  three  sites.  Factor  D  is  the 
average  score  of  questions  3,  4,  and  5.  Factor  D  score  was 
within  the  "slightly  satisfied"  range  (0  to  1)  . 


TABLE  4.7A 

ADMINISTRATIVE  SUPPORT  GROUP  ANOVA  TESTING 


QUEST  IONS /FAC TORS 

LOCATION  SITE  MEANSIU 

F  VALUE 

P  VALUE 

NSCORE S 

Hartley 
Fmaxl  2) 

CH 

Cl 

JX 

1 

RELATIONSHIP  WITH  MID  STAfF 

1  89 

2  19 

0  93 

7.22 

0.002 

0  97 

1.67 

2 

PROCESSING  OF  REQUESTS  FOR  CHANCE 

-1  28 

0  25 

-0  36 

5.04 

0.010 

0  97 

1.52 

3 

DECREE  OF  TRAINING 

-0  11 

0  50 

-0  21 

1  20 

0  308 

0  99 

1.74 

4  . 

USER  S  UNDERSTATING  OF  SYSTEM 

0  00 

1  33 

1  22 

2  24 

0.117 

0  96 

141 

5 

USER'S  FEELING  )F  PARTICIPATION 

0  89 

t  31 

0  19 

3.3b 

0.043 

0  99 

1  74 

6 

ATTITUDE  OF  MID  STAFF 

2  22 

2  42 

0  88 

13.B7 

0.000 

0  96 

1  85 

7. 

RELIABILITY  OF  OUTPUT 

0  06 

1  27 

0  93 

2  91 

0  064 

0  98 

1.52 

8 

RELEVANCY  OF  OUTPUT 

1 .00 

1  35 

0  64 

1  60 

0  212 

0  97 

1  44 

9 

ACCURACY  OF  OUTPUT 

-0  50 

1  44 

0.64 

7J5 

0  001 

0  98 

1  49 

10 

PRECISION  OF  OUTPUT 

-0  44 

1  08 

0  50 

5.17 

0.009 

0  98 

2  21 

1  1 

COMMUNICATION  WITH  MID  STAFF 

1  72 

2  17 

0  71 

•  ■55 

0.001 

0  97 

1  14 

12 

TIME  REQUIRED  FOR  NEW  DEVELOPMENT 

-1  06 

0  13 

-0.07 

1  83 

0  171 

0  99 

1  48 

13 

COMPLETENESS  Of  OUTPUT 

0  06 

1  04 

0  67 

1  56 

0  220 

0  98 

1  29 

OVERALL  SCORE 

4  44 

16  46 

6  57 

5.89 

0.005 

0  99 

1  68 

A 

MID  STAFF  AM)  SERVICES 

1  95 

2  26 

0  84 

12-09 

0.000 

0  97 

1  24 

B 

CONTRACTOR  SERVICES 

-1  17 

0  19 

-0  21 

4  34 

Q-oia 

0  99 

1  74 

C 

INFORMATION  OUTPUT 

0  03 

1  24 

0  68 

4  66 

0.014 

0  98 

1  56 

0 

KNOWLEDGE  AM)  INVOLVEMENT 

0  26 

1  05 

0  37 

2  04 

0  141 

0  99 

1  19 

NOTES 

(1)  cm  ■  nh  Charleston:  CL  •  mi  Camp 

(2)  hartley's  17)  approximate 

Lejeune 

critical 

JX  *  M4 
value  * 

jactsonvi 1 le 

3  30  at  a ipha 

x  0  05 

(8)  Significant  Administrative  Support  Group 
Findings 

Table  4.7b  represents  those  items  where  the 
ANOVA  testing  in  Table  4.7a  revealed  a  significant  difference 


60 


in  the  means  between  the  Administrative  Support  groups. 
Scheffe  multiple  comparison  testing  was  used  to  identify  the 
individual  differences  between  sites. 


The  Administrative  Support  group  at  the  NH 
Jacksonville  for  Factor  A  (MID  Staff  and  Services;  questions 
1,  6,  and  11)  were  less  satisfied  than  the  other  two  sites. 
The  NH  Camp  Lejeune  and  the  NH  Charleston  were  both 
essentially  within  the  "quite  satisfied"  range  (1  to  2), 
whereas  the  NH  Jacksonville  was  within  the  "slightly 
satisfied"  range  (0  to  1) . 

The  NH  Charleston  for  Factor  B  (Contractor 
Services;  questions  2  and  12)  was  less  satisfied  than  the 
other  two  sites.  The  NH  Camp  Lejeune  for  Factor  B  was  within 
the  "slightly  satisfied"  range  (0  to  l) ,  and  the  NH 
Jacksonville  was  within  the  "slightly  dissatisfied"  range  (0 
to  -1)  ,  but  was  not  statistically  different.  The  NH 
Charleston  for  Factor  B  was  within  the  "quite  dissatisfied" 
range  (-1  to  -2) . 

For  Factor  C  (Information  Product  Output; 
questions  7,  8,  9,  10,  and  13),  the  NH  Charleston  was  less 
satisfied  as  compared  with  the  NH  Camp  Lejeune,  however, 
scored  no  difference  with  the  NH  Jacksonville.  The 
significant  influence  to  this  finding  was  the  significant 
differences  paralleled  with  questions  9  and  10  dealing  with 
output  accuracy  and  precision.  Both  the  NH  Camp  Lejeune  and 
the  NH  Jacksonville  were  essentially  within  the  "slightly 
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satisfied"  range  (0  to  1)  for  Factor  C,  whereas,  the  NH 
Charleston  was  "neutral"  with  a  zero  score. 


TABLE  4.7B 

SCHEFFE  MULTIPLE  COMPARISON  TESTING;  ADMIN  SUPPORT  GROUP 


LOCATION  SITE  COMPARISONS 

QUEST  IONS /FACTORS 

CHARLESTON/ 
CAMP  LEJEUNE 

CHARLESTON/ 

JACKSONVILLE 

CAMP  LE  JEUNE/ 
JACKSONVILLE 

1 . 

RELATIONSHIP  WITH  MID  STAFF 

NS(I) 

NS 

S(  2) 

2. 

PROCESSING  OF  REQUESTS  FOR  CHANCES 

S 

NS 

NS 

5. 

USERS  FEELING  OF  PARTICIPATION 

NS 

NS 

S 

6 

ATTITUDES  OF  MID  STAFF 

NS 

S 

s 

9 

ACCURACY  OF  OUTPUT 

S 

NS 

NS 

10 

PRECISION  OF  OUTPUT 

s 

NS 

s 

1 1 

COMMUNICATION  WITH  MID  STAFF 

NS 

NS 

S 

OVERALL  SCORE 

s 

NS 

s 

A 

MID  STAFF  AM)  SERVICES 

NS 

S 

s 

B 

CONTRACTOR  SERVICES 

S 

S 

NS 

C. 

INFORMATION  OUTPUT 

S 

NS 

NS 

NOTES: 

(1  NS  »  Nonsignificant,  no  difference  in  weans 
( 2)  s  ■  Significant  difference  in  weans 

b.  The  Naval  Hospital  Charleston 

(1)  Differences  between  User  Groups 

Table  4.8a  shows  the  ANOVA  testing  results 
between  user  groups.  Significant  findings  (at  an  alpha  of 
0.05)  are  underlined.  The  Physician  group  sample  size 
consisted  of  21  participants;  the  Medical  Support  group 
consisted  of  71  participants;  and  the  Administrative  Support 
group  consisted  of  9  participants.  The  a  posteriori  testing 
for  normality  via  the  NSCORES  correlation  demonstrates  that 
the  data  has  a  normal  distribution.  The  disparity  in  the 
size  of  the  user  groups  sample  sizes  lead  to  reject 
homogeneity  of  variance  in  a  number  of  questions.  However, 
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on  subsequent  testing  using  nonparametric  procedures  revealed 
essentially  the  same  results  and  patterns. 


TABLE  4.8A 

NH  CHARLESTON  USER  GROUP  ANOVA  TESTING 


USER 

GROUP  ME ANSI  1 ) 

F  VALUE 

P  VALUE 

NSCORES 

Hartley 

QUESTIONS/FACTORS  ~ 

P 

M 

A 

CUKK LLA 

I . 

RELATIONSHIP  WITH  M<0  STAFF 

1  36 

1.74 

1  69 

0  84 

0  433 

0  98 

1  36 

2 

PROCESSING  OF  REQUESTS  FOR  CHANCE 

-0  45 

-0.42 

>1  28 

1  .08 

0  345 

0  99 

2.48 

3 

DEGREE  OF  TRAINING 

0  05 

0.62 

-0.11 

2.49 

0.068 

0  96 

1  96 

4. 

USER’S  UNDERSTATING  OF  SYSTEM 

0  14 

1.33 

0.00 

6  99 

0.001 

0  98 

1  39 

5 

USER  S  FEELING  OF  PARTICIPATION 

0.55 

1.25 

0.69 

1  70 

0  166 

0  97 

1  .46 

6 

ATTITUDE  OF  MID  STAFF 

1.57 

2  08 

2.22 

2  41 

0  095 

0  98 

1  31 

7 

RELIABILITY  OF  OUTPUT 

1  .07 

0  80 

0  06 

1  53 

0  221 

0.98 

3  99 

8 

RELEVANCY  OF  OUTPUT 

0  57 

1.06 

1.00 

1  02 

0  366 

0.98 

1  39 

9 

ACCURACY  OF  OUTPUT 

1  41 

1.31 

-0  50 

6.82 

0.002 

0.98 

2.59 

10 

PRECISION  OF  OUTPUT 

0.86 

0  58 

-0  44 

2  77 

0.066 

0  96 

3  11 

1 1 

COMMUNICATION  WITH  MID  STAFF 

1  36 

1  73 

1  72 

0  89 

0  413 

0  96 

1  16 

12 

TIME  REQUIRED  FOR  NEW  DEVELOPMENT 

-0  93 

0  10 

-1  06 

4.79 

O-OIO 

0  99 

1  37 

13 

COMPLETENESS  Of  OUTPUT 

0  52 

0  68 

0.06 

0  76 

0  459 

0  99 

3  98 

OVERALL  SCORE 

8  10 

13  06 

4  44 

3.22 

0.044 

0  99 

2  19 

A 

MID  STAFF  AT  SERVICES 

1  44 

1.85 

1  95 

1  58 

0  211 

0  98 

1  .  17 

B 

CONTRACTOR  SERVICES 

-0  69 

-0.  16 

-1  17 

3.19 

O.Q4S 

0  99 

3.76 

C 

INFORMATION  OUTPUT 

0  69 

0.69 

0.03 

2.23 

0  113 

0  98 

1  85 

D 

KNOWLEDGE  AT  INVOLVEMENT 

0  25 

1 .13 

0  26 

5.79 

0.004 

0  99 

1  22 

NOTES 

(1)  P  «  Physicians  m  ■  Medical  Support  a  « 

(2)  Hartley's  Fiaax(3  -,3,  approximate  critical 

Administrative  Support 
value  *  2  35  at  alpha  * 

0  05 

(2)  Significant  User  Group  Findings 

vable  4.8b  represents  those  items  where  the 
ANOVA  testing  in  Table  4.8a  revealed  a  significant  difference 
in  the  means  between  the  user  groups.  Scheffe  multiple 
comparison  testing  was  used  to  identify  the  individual 
differences  between  user  groups.  Physicians  were  less 
satisfied  when  compared  with  the  Medical  Support  group  for 
Factor  D  (User  Knowledge  and  Involvement) ;  no  difference  with 
the  Administrative  Support  group.  Physicians  and  the 
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Administrative  Support  group  were  within  the  "slightly 
satisfied"  range  (0  to  1)  .  Whereas,  the  Medical  Support 
group  was  within  the  "quite  satisfied"  range  (1  to  2)  .  Factor 
B  dealing  with  Contractor  services  (questions  2  and  12) ,  on 
multiple  comparison  testing  found  that  there  was  no 
significant  difference  between  the  groups.  All  groups  for 
Factor  B  were  essentially  within  the  "slightly  dissatisfied" 
range  (0  to  -1)  .  Again,  the  small  sample  size  of  the 
Administrative  Support  group  plays  a  role  in  the  resulting 
nonsignificant  findings. 

Interesting,  the  Administrative  Support  group 
were  less  satisfied  than  the  other  two  groups  as  to  the 
accuracy  of  the  output  (question  9) .  However,  there  were 
nonsignificant  differences  between  all  groups  found  in  Factor 
C  which  is  made  up  of  questions  7,  8,  9,  10,  and  13. 

TABLE  4.8B 

SCHEFFE  MULTIPLE  COMPARISON  TESTING;  NH  CHARLESTON 

USER  GROUPS 


USER  GROUPS  COMPARISON 

QUEST  IONS/ FACTORS 

PHYSICIAN/ 
MEDICAL  SUPPORT 

PHYSICIAN/ 

AOMIN  SUPPORT 

MEDICAL  SUPPORT/ 

ADMIN  SUPPORT 

4  USER'S  LM>ERSTAW>ING  Of  SYSTEM 

S(1) 

NS(2) 

NS 

9  ACCURACY  Of  OUTPUT 

NS 

S 

S 

1 2  T  1*1  REQUIRED  FOR  NEW  DEVELOPMENT 

S 

NS 

NS 

OVERALL  SCORE 

NS 

NS 

NS 

8  contractor  service* 

NS 

NS 

NS 

D  KNOWLEDGE  AND  INVOLVEMENT 

S 

NS 

NS 

NOTES 

(1)  S  *  significant  difference  In  weans 

(2)  ns  *  Nonsignificant:  no  difference  in  1 


c.  Naval  Hospital  Camp  Lejeune 

(1)  Differences  between  User  Groups 

Table  4.9a  shows  the  ANOVA  testing  results 
between  user  groups.  Significant  findings  (at  an  alpha  of 
0.05)  are  underlined.  The  Physician  group  sample  size 
consisted  of  22  participants;  the  Medical  Support  group 
consisted  of  75  participants;  and  the  Administrative  Support 
group  consisted  of  24  participants.  The  a  posteriori  testing 
for  normality  via  the  NSCORES  correlation  demonstrates  that 
the  data  has  a  normal  distribution.  Except  for  questions  6 
and  9,  the  Hartley  Fmax  test  revealed  homogeneity  of  variance 
between  groups.  However,  on  subsequent  testing  using 
nonparametric  procedures,  resulted  in  finding  essentially  the 
same  values  and  patterns. 

Although,  the  Physician  group's  mean  value  for 
Factor  B  (Contractor  Services;  questions  2  and  12)  was  within 
the  "slightly  dissatisfied"  range  (0  to  -1) ,  and  the  other 
two  groups  were  within  the  "slightly  satisfied"  range  of  (0 
to  1) ,  there  was  no  significant  difference  in  means  between 
the  groups.  There  was  no  significant  difference  between  the 
three  groups  concerning  the  information  product  output 
(Factor  C;  questions  7,  8,  9,  10,  and  13).  All  three  groups 
were  within  the  "quite  satisfied"  range  (1  to  2)  for  Factor 
C. 
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TABLE  4.9A 

NH  CAMP  LEJEUNE  USER  GROUP  ANOVA  TESTING 


USER 

GROUP  MEANS( 1 ) 

F  VALUE 

P  VALUE 

NSCORES 

Hartley 

fmax(2> 

QUEST  IONS /FACTORS  — 

P 

M 

A 

1 

RELATIONSHIP  WITH  MID  STAFF 

1.36 

1.22 

2.19 

5.76 

0.004 

0.96 

1  .76 

2 

PROCESSING  OF  REQUESTS  FOR  CHANCE 

-0.05 

0.24 

0.25 

0.37 

0.690 

0  98 

1  .25 

3 

DEGREE  OF  TRAINING 

0  05 

0.52 

0.50 

0.75 

0.475 

0.99 

1.07 

4. 

USER'S  UMDERSTAM5ING  OF  SYSTEM 

-0.59 

0.62 

1.33 

7.65 

0.001 

0  96 

1  .  17 

5 

USER'S  FEELING  OF  PARTICIPATION 

-0.21 

0.99 

1.31 

7.66 

0.001 

0.99 

1  .47 

6 

ATTITUDE  OF  MID  STAFF 

1.77 

1.33 

2.42 

7  09 

O  001 

0  98 

2.32 

7 

RELIABILITY  Of  OUTPUT 

1.32 

1  .27 

1.27 

0.01 

0.969 

0  96 

1.31 

a 

RELEVANCY  OF  OUTPUT 

0  71 

1.23 

1.35 

1.64 

0  199 

0.98 

1  30 

9 

ACCURACY  OF  OUTPUT 

1.30 

1.33 

1.44 

0.19 

0.628 

0.96 

2.65 

10 

PRECISION  OF  OUTPUT 

0  66 

0.79 

1  .06 

0  45 

0.637 

0  97 

1.54 

1  > 

COMMUNICATION  RUTH  MID  STAFF 

1.34 

1.03 

2.17 

6.59 

0-002 

0  98 

2  07 

12 

TIME  REQUIRED  FOR  NEW  DEVELOPMENT 

-0  52 

0  34 

0  13 

2  63 

0.076 

0  99 

1.50 

13 

COMPLETENESS  Of  OUTPUT 

0  04 

0.93 

1.04 

0  11 

0  897 

0  98 

1  41 

OVERALL  SCORE 

a  39 

12  08 

16  48 

253 

0  084 

0  99 

1.54 

A. 

MID  STAFF  Al*>  SERVICES 

1  49 

1  20 

2.26 

6.43 

0.000 

0  90 

1  43 

8 

CONTRACTOR  SERVICES 

-0  26 

0  29 

0.  19 

1.64 

0  199 

0.99 

1  13 

C 

INFORMATION  OUTPUT 

1  05 

! .  11 

1  24 

0  20 

0.823 

0  99 

1  19 

0 

KNOWLEDGE  AA t>  INVOLVEMENT 

-0  25 

0  76 

1  05 

6.61 

0.002 

0.99 

1.2! 

NOTES 

(u  p  «  Physicians:  m  *  Medical  support,  a  « 
(2)  Hartley  s  Faax0  3Q>  approximate  critical 

Administrative  Support 
value  *  2.24  at  alpha  « 

0.05 

(2)  Significant  User  Group  Findings 

Table  4 . 9b  represents  those  items  where  the 
ANOVA  testing  in  Table  4.9a  revealed  a  significant  difference 
in  the  means  between  the  user  groups.  Scheffe  multiple 
comparison  testing  was  used  to  identify  the  individual 
differences  between  user  groups. 

The  Medical  Support  group  was  less  satisfied 
when  compared  with  the  Administrative  Support  group  for  Factor 
A  (MID  Staff  and  Services;  guestions  1,  6,  11)  ,  and  no 
significant  difference  with  the  Physician  group.  The  Medical 
Support  group  and  the  Physician  group  for  Factor  A  were 
within  the  "quite  satisfied"  range  (1  to  2)  ,  whereas  the 
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Administrative  Support  group  was  within  the  "extremely 
satisfied”  range  (2  to  3)  .  The  Physician  group  was  less 
satisfied  for  Factor  D  (User  Knowledge  and  Involvement; 
questions  3,  4,  5)  than  the  two  other  groups.  The  Physician 
group  for  Factor  D  were  within  the  "slightly  dissatisfied” 
range  (0  to  -1) ,  whereas,  the  other  two  groups  were 
essentially  within  the  "slightly  satisfied"  range  (0  to  1) . 


TABLE  4.9B 

SCHEFFE  MULTIPLE  COMPARISON  TESTING;  NH  CAMP  LEJEUNE 

USER  GROUPS 


USER  GROUPS  COMPARISON 

QUEST  IONS/ FAC TORS 

PHYSICIAN/ 
MEDICAL  SUPPORT 

PHYSICIAN/ 
ADMIN  SUPPORT 

MEDICAL 

SUPPORT/  ADMIN 
SUPPORT 

1 

RELATIONSHIP  WITH  MID  STAFF 

NS<  1) 

NS 

S(  2) 

4 

USER'S  LMOE  RST AND 1 NC  OF  SYSTEM 

S 

S 

NS 

5 

USER’S  FEELING  OF  PARTICIPATION 

S 

S 

NS 

6 

ATTITUDES  OF  MID  STAFF 

NS 

NS 

S 

1 1 

COMMUNICATION  WITH  MID  STAFF 

NS 

NS 

s 

A 

MIS  STAFF  A»©  SERVICES 

S 

NS 

s 

D 

KNOWLEDGE  Ais©  INVOLVEMENT 

s 

S 

NS 

NOTES 

(1)  ns  •  Nonsiamt  icant  no  difference  in  weans 

(2)  s  »  Significant  difference  in  weans 

d.  Naval  Hospital  Jacksonville 

(1)  Differences  between  User  Groups 

Table  4 . 10a  shows  the  ANOVA  testing  results 
between  user  groups.  Significant  findings  (at  an  alpha  of 
0.05)  are  underlined.  The  Physician  group  sample  size 
consisted  of  36  participants;  the  Medical  Support  group 
consisted  of  61  participants;  and  the  Administrative  Support 
group  consisted  of  21  participants.  The  a  posteriori  testing 
for  normality  via  the  NSCORES  correlation  demonstrates  that 
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the  data  has  a  normal  distribution.  The  Hartley  Fmax  test 
revealed  homogeneity  of  variance  between  user  groups. 

There  was  no  significant  difference  between 
user  group  means  for  Factor  C  (Information  Product  Output; 
questions  7,  8,  9,  10,  and  13) .  All  user  groups  were  within 
the  "slightly  satisfied"  range  (0  to  1)  for  Factor  C. 
Question  3  dealing  with  the  degree  of  training  was  found 
initially  to  have  significant  differences  in  user  group 
means.  However,  on  subsequent  testing  as  denoted  in  Table 
4.10b,  there  was  no  significant  difference.  All  user  groups 
for  question  3  were  essentially  within  the  "slightly 
dissatisfied”  range  (0  to  -1) . 
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TABLE  4.10A 

NH  JACKSONVILLE  USER  GROUP  ANOVA  TESTING 


QUEST  IONS /FACTORS 

USER 

CROUP  MEANS ( !> 

F  VALUE 

P  VALUE 

NSC ORES 

Hartley 

f*ax<2) 

P 

ft 

A 

1  1 V1* 

1 . 

RELATIONSHIP  WITH  ftJD  STAFF 

0.29 

0.94 

0.93 

3.70 

0.028 

0.99 

1.44 

2 

PROCESSING  OF  REQUESTS  FOR  CHANGE 

-1.58 

-0.76 

-0.36 

5.18 

8.007 

0  99 

2.19 

3. 

DEGREE  OF  TRAINING 

-1.08 

-0.30 

-0.21 

3.29 

0.041 

0.99 

1.58 

4. 

USER'S  UFDERSTAKDING  OF  SYSTEft 

-0.14 

0.77 

1.12 

5  80 

0-004 

0.98 

1.18 

5. 

USERS  FEELING  OF  PARTICIPATION 

-0.44 

0.53 

0.  19 

4.81 

0.010 

0.99 

2.00 

6 

ATTITUDE  OF  MID  STAFF 

0.46 

0.85 

0.88 

1.34 

0.265 

0.98 

1.89 

7 

RELIABILITY  OF  OUTPUT 

0.99 

0.81 

0  93 

0.17 

0.846 

0.97 

1 .  12 

S 

RELEVANCY  OF  OUTPUT 

0.54 

0.76 

0.64 

0.26 

0.772 

0.98 

1.07 

9. 

ACCURACY  OF  OUTPUT 

1  . 17 

091 

0.64 

1.02 

0.364 

0.97 

1.51 

10 

.  PRECISION  OF  OUTPUT 

1.04 

0.79 

0.50 

1.34 

0.266 

0.99 

1  81 

11 

COMMUNICATION  WITH  MID  STAFF 

0  13 

0.68 

0  71 

2.61 

0.078 

0.99 

2.01 

12 

.  TIME  REQUIRED  FOR  NEW  DEVELOPMENT 

-1.42 

-0  36 

-0.07 

6  08 

0.003 

0.99 

1.40 

13 

COMPLETENESS  OF  OUTPUT 

0.17 

067 

0.67 

1  .  15 

0.319 

0.97 

1.64 

OVERALL  SCORE 

0  11 

6.30 

6.57 

3.19 

0.04S 

0.99 

2.22 

A  . 

MID  STAFF  AND  SERVICES 

0.29 

0.83 

0.84 

2.96 

0.056 

0.98 

1  .74 

B 

CONTRACTOR  SERVICES 

-1.50 

-0.56 

-0.21 

7.27 

O.OOI 

0.99 

2  09 

C 

INFORMATION  OUTPUT 

0  78 

0.79 

0  68 

0  07 

0.931 

0.98 

1 .39 

D 

KNOWLEDGE  AM)  INVOLVEMENT 

0.56 

0.33 

0.37 

6.50 

0.002 

0.99 

2.09 

NOTES 

n)  P  •  Physicians:  ft  ■  Medical  support,  A  ■  Administrative  support 

(2)  Hartley  s  Fmax(3  3a)  approxl«ate  critical  value  »  2.25  at  alpha  «  0.05 


(2)  Significant  User  Group  Findings 

Table  4 . 10b  represents  those  items  where  the 
ANOVA  testing  in  Table  4.10a  revealed  a  significant  difference 
in  the  means  between  the  user  groups.  Scheffe  multiple 
comparison  testing  was  used  to  identify  the  individual 
differences  between  user  groups. 

The  Physician  group  was  significantly  less 
satisfied  for  Factor  B  (Contractor  Services;  questions  2  and 
12)  than  the  other  two  user  groups.  The  Medical  Support 
group  and  the  Administrative  Support  group  for  Factor  B  were 
within  the  "slightly  dissatisfied"  range  (0  to  -1)  whereas, 
the  Physician  group  was  within  the  "quite  dissatisfied"  range 


(-1  to  -2) .  The  Physician  group  was  significantly  less 
satisfied  for  Factor  D  (User  Knowledge  and  Involvement; 
questions  3,  4,  and  5)  than  the  two  other  user  groups.  The 
Medical  Support  group  and  the  Administrative  Support  group  for 
Factor  D  were  within  the  "slightly  satisfied"  range  (0  to  1) 
whereas,  the  Physician  group  was  within  the  "slightly 
dissatisfied"  range  (0  to  -1) . 


TABLE  4.10B 

SCHEFFE  MULTIPLE  COMPARISON  TESTING;  NH  JACKSONVILLE 

USER  GROUPS 


USER  GROUPS  COMPARISON 

QUEST  IONS/ FACTORS 

PHYSICIAN/ 

MEDICAL  SUPPORT 

PHYSICIAN/ 

ADMIN  SUPPORT 

MEDICAL  SUPPORT/ 
ADMIN  SUPPORT 

1  RELATIONSHIP  WITH  MIO  STAFF 

S(1) 

NS<2> 

NS 

2  PROCESSING  Of  REQUESTS  FOR  CHANGES 

S 

S 

NS 

3  DEGREE  OF  TRAINING  PROVIDED 

NS 

NS 

NS 

4  USER  S  M€RSTAKt>INC  Of  SYSTfm 

s 

S 

NS 

5  USER  S  FEELING  OF  PARTICIPATION 

S 

NS 

NS 

12  T U»E  REQUIRED  FOR  NEW  DEVELOPMENT 

s 

S 

NS 

OVERALL  SCORE 

s 

s 

NS 

B  CONTRACTOR  SERVICES 

s 

s 

NS 

D  KNOWLEDGE  AND  INVOLVEMENT 

s 

s 

NS 

NOTES 

<i)  s  *  Significant  difference  in  weans 

(2)  ns  •  Nonsignificant:  no  difference  In  weans 

2.  Time  of  Use  Correlation 

Table  4.11  shows  the  breakdown  of  the  three  hospital 
data  for  correlation  as  well  as  trend  analysis  mid-points  of 
user's  level  of  satisfaction  with  time  of  use  of  the  CHCS. 
User  time  of  use  of  the  CHCS  was  compared  against  each  of 
the  four  factors  (MID  Staff  and  Services,  Contractor 
Services,  Information  Product  Output,  and  User  Knowledge  and 
Involvement) .  Significant  findings  are  underlined. 
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TABLE  4.11 

TIME  OF  USE  CORRELATION  AND  TRENDS 
(COMBINED  THREE  HOSPITAL  DATA) 


FACTOR /CROUP 

CORRELATION 

TRENDS  BY  6  MONTH  INTERVALS 

MEAN  SCORES 

PEARSON  S 
CORRELATION 
COEFFICIENT  (f) 

RH0  TEST 

P  VALUE 

bb 

6-11 

12-17 

18+ 

FACTOR  A: 

ALL  CROUPS 

-0  016 

0.38 

1.29 

1.31 

1.39 

1  .  14 

SAMPLE  SIZE 

340 

— 

67 

98 

81 

94 

PERCENT  Of 

— 

— 

20% 

29% 

23% 

28% 

PHYSICIANS 

-0.080 

0.24 

0.94 

1 .13 

0.81 

0.76 

SAMPLE  SI7E 

79 

18 

25 

19 

17 

PERCENT  OF 

— 

— 

23% 

32% 

24% 

21% 

MEDICAL  SUPPORT 

-0  084 

Oil 

1.45 

1.27 

1.55 

1.03 

SAMPLE  SIZE 

207 

-- 

39 

60 

52 

56 

PERCENT  OF 

— 
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AOMIN  SUPPORT 

♦0  268 

♦  0  03 
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♦0.44 

PHYSICIANS 
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♦0.014 

♦0.42 

♦0  89 

♦0.90 

♦0.60 

AOMIN  SUPPORT 

♦0.120 

♦  0  19 

mam 

♦  1  17 

♦  0.32 

♦0.75 

The  Administrative  Support  group  demonstrated  a 
significant  positive  correlation  (Pearson's  sample  correlation 
coefficient  r-.2l ,  P  value=0.03)  for  time  of  use  of  the  CHCS 
and  Factor  C  (Information  Product  Output) .  These  findings 
suggest  that  as  the  time  of  use  increases,  the  Administrative 
Support  group's  level  of  satisfaction  with  the  information 
product  output  increases.  There  were  no  other  significant 
correlations  noted. 

Table  4.11  shows  the  breakdown  of  the  three  hospital 
data  for  trend  analysis  of  user's  level  of  satisfaction  at 


six  month  interval  time  periods  of  use  of  the  CHCS.  A  word 
of  caution  about  trend  analysis.  The  trend  analysis  points 
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are  derived  from  the  mean  of  the  respective  scores  in  each 
six  month  time  series  interval.  One  form  of  time  series 
trend  analysis  attempts  to  fit  a  straight  line  (linear 
relationship  trend  line)  between  these  points,  and  is 
generally  used  in  forecasting.  As  such,  these  findings  are 
only  suggestive  and  based  on  observable  straight  line  fitting 
of  the  data.  The  six  month  intervals  used  to  look  at  trends 
are  fairly  evenly  distributed  for  each  of  the  groups  so  as  to 
limit  a  weighting  problem. 

a.  Factor  A  (MID  Staff  and  Services)  Trends 

Figure  4 . 6  shows  the  plotting  of  the  six  month 
mean  scores  by  All  Groups  combined  as  well  as  each  of  the 
individual  user  groups  for  Factor  A  (MID  Staff  and  Services) 
trends.  Although  all  scores  demonstrate  at  least  within  the 
"slightly  satisfied"  range  (0  to  1),  the  Physician  and 
Medical  Support  user  groups  appear  to  have  a  downward  trend. 
This  suggests  that  as  Physician  and  Medical  Support  personnel 
increase  the  time  of  use  of  the  system,  their  level  of 
satisfaction  with  the  local  MID  staff  and  services 
diminishes;  whereas,  the  Administrative  Support  user  group 
appears  to  have  an  upward  trend.  This  suggests  that  as 
Administrative  Support  personnel  increase  the  time  of  use  of 
the  system,  their  level  of  satisfaction  with  the  local  MID 
staff  and  services  increases. 
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TREND  ANALYSIS 


Factor  A  (MID  Staff  and  Services)  Satisfaction  Scores 


Figure  4.6.  Trend  Analysis  Factor  A 


b.  Factor  B  (Contractor  Services)  Trends 

Figure  4.7  shows  the  plotting  of  the  six-month 
mean  scores  by  All  Groups  combined  as  well  as  each  of  the 
individual  user  groups  for  Factor  B  (Contractor  Services) 
trends.  The  Physician  and  Medical  Support  user  groups  appear 
to  have  a  downward  trend.  This  suggests  that  as  Physician 
and  Medical  Support  personnel  increase  the  time  of  use  of 
the  system,  their  level  of  satisfaction  with  the  contractor 
services  (i.e.,  changes  to  the  system)  diminishes.  The 
Physician  and  Medical  Support  user  groups  make  up  a  large 


73 


percentage  of  users  in  this  study,  and  this  contributes  to 
the  very  noticeable  downward  trend  for  the  All  Group 
category. 

TREND  ANALYSIS 


Factor  B  (Contractor  Services)  Satisfaction  Scores 


Figure  4.7.  Trend  Analysis  Factor  B 


c.  Factor  C  (Information  Product  Output)  Trends 

Figure  4.8  shows  the  plotting  of  the  six-month 
mean  scores  by  All  Groups  combined  as  well  as  each  of  the 
individual  user  groups  for  Factor  C  (Information  Product 
Output)  trends.  The  Medical  Support  user  group  appears  to 
have  a  downward  trend.  This  suggests  that  as  Medical  Support 
personnel  increase  the  time  of  use  of  the  system,  their 
level  of  satisfaction  with  the  information  product  output 
diminishes.  As  expected,  the  Administrative  Support  user 
group  appears  to  have  a  sharp  upward  trend  (this  factor  also 
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had  a  positive  correlation) .  This  suggests  that  as 
Administrative  Support  personnel  increase  the  time  of  use  of 
the  system,  their  level  of  satisfaction  with  the  information 
product  output  increases.  The  Physician  user  group  appears  to 
have  an  upward  trend  up  through  the  first  17  months  of  use, 
and  then  thereafter  drops. 


TREND  ANALYSIS 

Factor  C  (Information  Product  Output)  Satisfaction  Scores 
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Figure  4.8.  Trend  Analysis  Factor  C 
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d.  Factor  D  (User  Knowledge  and  Involvement)  Trends 

Figure  4.9  shows  the  plotting  of  the  six-month 
mean  scores  by  All  Groups  combined  as  well  as  each  of  the 
individual  user  groups  for  Factor  D  (User  Knowledge  and 
Involvement)  trends.  This  factor  deals  with  an  individual's 
perceptions  of  the  training  provided,  understanding  the 
system,  and  the  degree  the  individual  felt  they  were 


75 


participants  in  the  system's  development.  The  Physician  user 
group  appears  to  have  a  downward  trend.  This  suggests  that 
as  physicians  increase  the  time  of  use  of  the  system,  their 
level  of  satisfaction  with  understanding  and  being  able  to 
use  the  system  diminishes.  The  Administrative  Support  user 
group  appears  to  have  an  upward  trend.  This  suggests  that  as 
administrative  support  personnel  increase  the  time  of  use  of 
the  system,  their  level  of  satisfaction  with  understanding 
and  being  able  to  use  the  system  increases. 
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V.  ANALYSIS  OF  FINDINGS 


A.  INTRODUCTION 

It  became  apparent  while  conducting  this  study  that  the 
Baroudi  and  Orlikowski  [2]  short-form  UIS  instrument  was 
inadequate  to  assess  the  overall  user  satisfaction  level  for 
the  CHCS  in  this  setting.  As  a  result,  the  significance  and 
reporting  of  the  overall  UIS  scores  have  been  purposely 
diminished.  Rather,  the  reporting  and  analysis  of  the 
findings  is  more  appropriately  contained  within  the  four 
factors  found  to  summarize  the  13  questions  asked  of  the 
participants.  As  with  any  exploratory  study,  this  small 
sampling  is  not  intended  to  provide  definitive  results. 
Instead,  the  findings  here  are  intended  to  provide  insight 
and  basis  for  further  investigation.  In  discussing  the 
findings,  the  intention  will  be  to  deal  with  more  global 
issues. 

In  the  first  section,  the  homogeneity  of  the  sample, 
responses,  and  structure  of  the  short- form  UIS  instrument  is 
discussed.  This  is  important  as  stated  earlier  in  this  paper 
for  it  forms  the  foundation  to  be  able  to  make  the  statements 
regarding  the  stability  of  the  four  factor  structure,  and  in 
discussing  differences  without  the  influence  of  instrument 
bias.  In  the  second  section,  the  limitations  confronting  the 
current  short-form  UIS  instrument  and  its  impact  on  the 
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overall  assessment  of  UIS  is  discussed.  Thirdly,  the  survey 
findings  of  the  three  hospital  data  sampling  will  be 
discussed  for  distinctive  differences  and  similarities 
between  the  three  user  groups.  Lastly,  the  effect  of  length 
of  time  of  use  of  the  system  on  user  satisfaction  levels  for 
the  four  factors  will  be  discussed. 

B.  HOMOGENEITY  OF  SAMPLE  AND  RESPONSES 

As  expected,  the  sample  data  revealed  a  relatively 
homogenous  group.  The  ANOVA  testing  essentially  supports  the 
homogeneity  of  the  sample.  Although  there  existed  a 
significant  difference  in  the  median  education  level  at 
Jacksonville  ("bachelor  degree"  vs  "some  college"  at  the 
other  two  sites)  on  subsequent  testing  for  correlation  of 
education  and  the  level  of  satisfaction  revealed  no 
significant  correlation.  Igbaria  and  Nachman  [5]  in  their 
study  found  similar  findings  in  that  education  was  not 
correlated  with  user  satisfaction.  The  same  can  be  said 
essentially  for  the  time  of  use  of  the  system  which  will  be 
amplified  on  later. 

The  homogeneity  is  further  supported  when  equivalence 
testing  for  covariance  and  factor  structure  of  the  short-form 
UIS  instrument  responses  was  conducted.  The  finding  of 
invariance  between  the  three  sites  (subpopulations)  for:  1) 
covariance,  2)  four-factor  structure,  3)  factor  loadings  and 
pattern,  4)  error/uniqueness,  and  5)  factor  variance  and 
covariances,  demonstrates  the  stability  of  the  new  structure, 
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eliminates  concerns  about  construct  bias,  and  enhances  the 
argument  for  homogeneity  of  the  sample. 

C.  LIMITATIONS  WITH  THE  CURRENT  SHORT- FORM  UIS  INSTRUMENT 
1.  Factor  Structure 

The  fundamental  change  of  the  current  short-form  UIS 
instrument  from  a  three-factor  construct  to  a  four-factor 
construct  seriously  jeopardized  its  reliability  and  validity. 
The  four-factor  solution  was  tested  and  confirmed  to  be  the 
stable  form  in  this  specific  case.  Interesting,  the  Ives  et 
al.[3],  22  question  survey  construct  factored  into 
essentially  the  same  four-factor  identifiers  (MID  Staff  and 
Services,  Contractor  Services,  Information  Product  Output, 
and  User  Knowledge  and  Involvement)  as  found  in  this  study. 
Ives  et  al.[3],  eliminated  the  "vendor  support"  factor  due  to 
it  only  having  one  item  loading  into  it. 

Back  when  the  Ives  et  al.[3]  UIS  instrument  construct 
was  developed,  and  later  reduced  to  the  current  13  question 
construct  by  Baroudi  and  Orlikowski  [2],  less  emphasis  was 
given  to  systems  contracted  out  to  external  vendors.  In  the 
Department  of  Defense,  more  and  more  large-scale  information 
systems  development  is  being  contracted  out,  as  is  the  case 
with  the  CHCS . 

With  the  four-factor  construct  there  exists  an 
internal  reliability  problem.  Factor  B  (Contractor  Services) 
suffers  much  the  same  problem  encountered  with  the  Ives  et 
al.[3]  study,  in  that  too  few  items  make  up  the  factor.  The 


Cronbach  alpha  for  Factor  B  was  0.68  which  is  below  the 
minimum  internal  reliability  of  0.70  that  Nunnally  [27] 
recommends.  The  rule  of  thumb  for  the  number  of  items  (or 
variables)  per  factor  is  that  there  must  be  at  least  three 
distinct  variables  [23] [27].  Adding  the  question  concerning 
vendor  support  from  the  Ives  et  al.  [3]  study  would  give 
Factor  B  the  minimum  three  variables. 

2.  Weighting  of  Factors 

The  overall  UIS  score  for  the  current  short-form  UIS 
instrument  is  determined  by  the  sum  of  the  13  question's 
averaged  scores.  This  is  the  correct  procedure,  however, 
unless  each  factor  has  the  same  number  of  questions  (or 
variables)  the  overall  UIS  score  is  biased  to  that  factor (s) 
with  the  most  variables.  The  current  short-form  UIS 
instrument  is  heavily  influenced  by  Factor  C  (Information 
Product  Output)  which  has  five  questions  making  up  this 
factor.  Looking  at  the  questions  that  make  up  Factor  C, 
there  exists  redundancy.  Questions  9  (Accuracy)  and  10 
(Precision)  could  be  well  served  by  question  7  (Reliability) . 
A  definition  of  question  7  could  be  offered  to  mean  that  the 
information  product  output  is  reliable,  timely,  accurate,  and 
precise.  The  same  holds  for  question  8  (Relevancy)  which 
could  be  well  served  by  question  13  (Completeness)  .  This 
would  reduce  the  emphasis  on  product  output  to  two  questions. 
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3.  Information  Product  Output  Factor 

As  Factor  C's  name  implies,  Information  Product  Output 
only  deals  with  the  system's  outputs.  An  important  assessment 
in  dealing  with  user's  perception  of  a  system  needs  to 
address  the  user  interface.  If  the  system  has  a  difficult 
user  interface  it  is  more  likely  that  the  user  will 
understand  the  system  less,  and  require  more  training  in 
order  for  the  user  to  become  accustomed  to  the  machine.  The 
term  "user-friendly"  is  used  to  denote  a  user  interface  that 
is  logically  designed  with  the  user's  needs,  intuitively 
obvious,  flexible,  and  offers  on-line  help.  After  talking 
with  some  of  the  users  and  seeing  the  system  in  operation, 
the  user  interface  is  an  area  of  concern  in  this  case.  The 
user  interface  is  an  area  not  addressed  by  the  current 
short-form  UIS  instrument.  Feedback  from  users  noted  this 
omission.  Interesting,  the  Ives  et  al.[3]  survey  had  user 
interface  (convenience  of  access)  which  had  very  high 
reliability  correlations  and  factor  loaded  into  their 
four-factor  solution.  Combining  the  user  interface  question 
along  with  the  two  output  directed  questions  from  above, 
would  give  Factor  C  three  variables  and  equality  with  the 
other  factors. 

It  should  be  noted  that  Factor  D  may  be  considered  an 
indirect  measure  of  "user-friendliness."  Factor  D  deals  with 
user's  assessment  of  training,  understanding,  and  involvement 


with  the  system  development.  This  issue  will  be  further 
discussed  in  the  next  section. 


D.  SIGNIFICANT  FACTOR  DIFFERENCES  BY  USER  GROUPS 


Figure  5.1  shows  the  measure  of  user  satisfaction  by  each 
of  the  four  factors  and  the  three  user  groups. 


Figure  5.1.  Measure  of  User  Satisfaction 

I.  Factor  A  (MID  Staff  and  Services) 

Each  of  the  user  groups  was  satisfied  with  the  local 
MID  staff  and  services.  This  factor  assesses  the  user's 
perception  toward  their  relationship  and  the  ability  to 
communicate  with  the  Management  Information  Department  (MID) . 
Although  each  of  the  user  groups  knows  that  the  system  is 
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being  designed  and  implemented  by  an  external  vendor,  all 
problems,  change  requests,  and  training  is  first  handled 
through  the  local  MID.  The  level  of  satisfaction  tends  to  go 
up  as  the  level  of  system  interaction  complexity  goes  down. 
That  is  to  say,  physicians  were  significantly  less  satisfied 
than  the  other  two  groups  because  their  system  interaction  is 
the  greatest  and  hence  the  greatest  interaction  with  the 
local  MID.  Next  is  the  Medical  Support  group,  followed  by 
the  Administrative  Support  group  with  the  least  system 
routines  to  interact  with.  Figure  5.1  shows  this  stepping 
stone  cascade  effect. 

2.  Factor  B  (Contractor  Services) 

Each  of  the  user  groups  was  dissatisfied  with  the 
contractor's  services.  This  factor  assesses  the  user's 
perception  toward  the  timeliness  of  incorporating  changes  to 
the  system.  The  bureaucratic  process  for  change  in  government 
contractual  agreements  can  be  lengthy.  Bailey  and  Pearson 
[18]  noted  that  the  variables  in  this  factor  were  ones  that 
were  the  most  frequently  cited  as  causing  dissatisfaction. 
With  apparently  all  three  user  groups  wanting  the  system  to 
change,  it  brings  to  question  the  adequacy  of  the  initial 
system's  analysis  and  requirements  process.  To  design  a 
system  "on  the  fly"  through  prototype-like  software  version 
updates,  can  lead  to  cost  overruns  and  delays  in  delivery. 

As  mentioned  above,  Factor  B's  reliability  alpha  is 
below  the  minimum  level  desired  for  exploratory  research. 


Some  caution  is  warranted.  However,  as  can  be  seen  in  Figure 
4.5  and  questions  2  and  12  that  make  up  Factor  B,  the  results 
for  Factor  B  are  a  good  reflection  of  the  user  groups 
perceptions  (no  equally  counter-balancing  responses) . 

3.  Factor  C  (Information  Product  Output) 

Each  of  the  user  groups  was  satisfied  with  the 
information  that  the  system  produced.  In  fact,  there  is  no 
significant  difference  between  the  three  user  groups  as  to 
their  level  of  satisfaction  as  depicted  in  Figure  5.1.  This 
factor  asked  the  user's  assessment  of  the  system's 
information  output  for  reliability,  relevancy,  accuracy, 
precision,  and  completeness. 

4.  Factor  D  (User  Knowledge  and  Involvement) 

This  factor  assesses  the  user's  perception  of  the 
training  received,  their  understanding  of  the  system 
regarding  the  ability  to  use  it,  and  their  feeling  of 
participation  with  the  system's  development.  Physicians  were 
significantly  less  satisfied  than  the  two  other  groups.  The 
Physicians  were  dissatisfied  versus  the  Medical  Support  and 
Administrative  Support  groups  reporting  being  satisfied.  The 
system  is  designed  to  be  driven  by  the  Physician.  As  such, 
the  Physician  is  given  the  widest  access  to  the  system's 
capabilities,  and  has  the  most  incentive  for  wanting  a  user- 
friendly  system.  Unfortunately,  the  myriad  of  menus  with 
limited  on-line  help  makes  the  system  in  its  current 
configuration  difficult  to  use.  The  short-form  UIS 
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instrument  used  in  this  study  as  already  mentioned,  does  not 
directly  assess  the  user  interface.  More  training  would  only 
be  a  short-term  resolve.  Recurring  training  is  difficult  to 
schedu  and  would  require  additional  resources,  adding  to 
the  cost  of  overhead  maintenance  for  this  system  (i.e.,  for 
trainers,  space  for  training,  and  time  away  from  health  care- 
related  productivity).  Strassmann  [44]  contends  that  sending 
users  to  classes,  especially  if  this  done  entirely  on  the 
organization's  time  and  at  the  organization's  training 
center,  is  about  the  most  expensive,  least  effective  way  of 
promoting  the  desired  learning.  He  supports  that  the  system 
should  fit  people,  not  the  other  way  around  [44]. 

E.  TIME  OF  USE  CORRELATION 

The  assumption  held  is  that  as  the  length  of  time  of  use 
of  the  system  increased,  the  user's  level  of  satisfaction 
would  increase.  This  assumption  is  made  in  the  belief  that 
the  user  would  realize  with  time  and  experience  the  benefits 
and  efficiencies  that  the  system  should  impart.  The  only 
significant  correlation  was  for  the  Administrative  Support 
group  with  Factor  C  (Information  Product  Output) .  The 
Administrative  Support  group  exhibited  a  positive  correlation 
with  the  system's  output.  There  were  no  other  significant 
correlations  noted.  This  lack  of  correlation  for  the  most 
part  may  be  significant.  It  does  say  that  a  new  user's  level 
of  satisfaction  for  that  respective  factor  is  essentially 
(statistically  speaking)  the  same  as  a  long-time  user. 
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VI .  CONCLUSION 


The  Composite  Health  Care  System  (CHCS)  is  a  billion 
dollar  military  hospital  information  system  venture  currently 
being  undertaken  by  the  Department  of  Defense.  The  design 
and  implementation  of  the  CHCS  is  under  contract  to  an 
external  vendor.  Good  management  practices  deem  assessment 
of  effectiveness  of  any  implementation  effort.  One  of  the 
most  commonly  used  approaches  to  assess  effectiveness  of  a 
computer-based  information  system  is  the  surrogate  measure  of 
user's  attitudes. 

In  this  study,  a  commonly  used,  and  previously  validated 
13  question  short-form  user  information  satisfaction  (UIS) 
instrument  developed  by  Baroudi  and  Orlikowski  [2],  was  used 
to  assess  its  ability  to  assess  overall  UIS  for  the  CHCS. 
Baroudi  and  Orlikowski  [2,  p.  55]  make  it  clear  "that  the 
short-form  measure  is  not  a  universally  applicable  and 
immutable  measure.  It  thus  may  be  appropriate  in  various 
situations  to  modify  the  measure  to  more  adequately  reflect 
the  requirements  of  the  specific  organization." 

One  of  the  important  findings  in  this  study  is  that  the  13 
question  short-form  UIS  instrument  is  inadequate  to  assess  the 
overall  satisfaction  of  the  CHCS.  However,  the  information 
gained  from  its  use,  combined  with  further  investigation  can 
assist  in  the  evaluation  of  effectiveness  of  the  CHCS.  The 
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use  of  a  short-form  UIS  instrument  can  be  a  useful  tool. 
When  properly  designed,  it  offers  the  information  systems 
manager  a  quick  and  easy  tool  to  assess  areas  of  potential 
problems  for  further  investigation. 

A.  CONCLUSIONS 

1.  Validation  of  the  Short-Form  UIS  Instrument 

As  mentioned,  the  most  important  conclusion  is  that 
the  13  question  short-form  instrument  is  inadequate  for 
evaluation  of  overall  UIS.  Limitations  associated  with  a 
fundamental  change  in  the  factor  structure  due  to  the 
significant  impact  of  contractor  services,  unevenly 
distributed  variables  in  a  couple  of  factors,  low  internal 
consistency  coefficient  in  one  factor,  no  assessment  of  user 
interaction  with  the  system,  and  inconsistent  convergent 
validity  findings  all  contributed  to  the  inadequacy  of  this 
instrument  for  the  evaluation  of  overall  UIS  of  the  CHCS. 

2.  Areas  of  Satisfaction 

All  three  user  groups  (Physicians,  Medical  Support, 
and  Administrative  Support)  were  satisfied  with  the  local 
Management  Information  Department  (MID)  staff  and  services, 
as  well  as  the  information  product  output. 

The  Medical  Support  and  the  Administrative  Support 
groups  were  satisfied  with  the  training,  understanding  of  the 
system,  and  level  of  participation. 
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3.  Areas  of  Dissatisfaction 

All  three  user  groups  were  dissatisfied  with  the 
contractor  services  to  accomplish  change  to  the  system. 
Physicians  were  dissatisfied  with  the  training,  understanding 
of  the  system,  and  the  level  of  participation.  Additionally, 
Physicians  were  statistically  less  satisfied  than  the  other 
two  user  groups  for  local  MID  staff  and  services,  contractor 
services,  and  for  the  training,  understanding  of  the  system, 
and  level  of  participation. 

4.  Time  of  Use  Correlation 

The  only  significant  correlation  was  the  positive 
correlation  exhibited  by  the  Administrative  Support  group 
between  the  length  of  time  of  CHCS  use  and  the  information 
product  output.  There  were  no  other  significant  correlations 
noted . 

B.  FUTURE  RESEARCH 

1.  Re-design  of  the  Short- form  UIS  Instrument 

Based  on  the  findings  from  this  study,  a  12  question 
(or  variable)  short-form  UIS  instrument,  Appendix  G,  is 
suggested  for  future  research  investigation.  In  the 
demographics  section,  questions  concerning  education,  age, 
gender,  and  previous  computer  use  should  be  considered  for 
elimination.  Each  of  these  user  attributes  have  been  found 
in  a  previous  study  [5],  and  for  the  most  part  supported  by 
this  study,  not  to  correlate  with  UIS.  In  the  questionnaire 
section,  the  use  of  an  example  to  illustrate  the  scale 
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positions  and  their  meanings  would  be  highly  beneficial  for 
clarity.  Baroudi  and  Orlikowski  [2]  offer  a  good  example  for 
use.  A  six  interval  Likert-type  measurement  scaling  model 
vice  the  current  seven  interval  model  should  be  considered. 
Using  a  six  interval  model  eliminates  "neutral"  responses 
without  changing  the  scoring  of  the  intervals  (-3  to  +3) . 

Referring  to  Appendix  G,  each  of  the  four  factors 
would  have  three  questions  associated  with  it.  Factor  A  (MID 
Staff  and  Services)  would  be  unchanged  and  be  represented 
with  the  same  questions  (in  this  case,  questions  1,  6,  and 
10) .  ictor  B  (Contractor  Service)  would  consist  of  the  same 
questions  plus  the  added  question  about  vendor  support 
(questions  2,  8,  and  11).  Factor  C  (Information  Input  and 
Product)  would  consist  of  questions  7,  12,  and  the  added 
question  about  "convenience  of  access"  (question  9)  .  Factor 
D  (User  Knowledge  and  Involvement)  would  be  unchanged  using 
questions  3,  4,  and  5.  Having  equally  distributed  variables 
for  each  factor  will  prevent  any  one  factor  from  biasing  the 
overall  score  (which  will  now  range  from  -36  to  +36  vice  -39 
to  +39  due  to  one  less  question) . 

Clarity  of  each  question  is  important.  Especially  for 
the  variable  "convenience  of  access"  where  the  emphasis  is 
looking  at  the  ease  or  difficulty  with  which  the  user  may  act 
to  utilize  the  capability  of  the  system.  Bailey  and  Pearson 
[18]  offer  complete  definitions  for  each  of  the  questions. 
All  of  the  current  procedures  described  by  Baroudi  and 


Orlikowski  [2]  for  disseminating  the  instrument  and  ensuring 
confidentiality  should  be  used. 

a.  Validation  of  the  Mew  Short-form  Instrument 

A  large  sampling  of  multiple  hospitals  (such  as  in 
this  study)  using  the  new  short-form  instrument  will  be 
necessary  in  order  to  validate  the  instrument.  In  addition 
to  the  methods  used  in  this  study  (i.e.,  exploratory  and 
confirmatory  factor  analysis,  internal  consistency  testing, 
and  equivalence  testing) ,  alternative  forms  of  parallel  tests 
(i.e.,  correlation  with  OT&E  survey  findings)  or  the 
split-half  approach  for  estimates  of  reliability  as  described 
by  Nunnally  [27]  should  be  used.  Nunnally  [27]  does  not 
recommend  the  retest  method  as  an  estimate  of  reliability  as 
previous  researchers  have  suggested.  In  addition,  include 
sufficient  time  to  allow  a  larger  number  of  user  interviews 
(minimum  sample  size  of  20)  for  all  three  groups  in  order  to 
conduct  statistically  meaningful  convergent  validity 
correlation  study. 
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APPENDIX  A 


Part  A:  General  Information 

1.  Hospital  Department:  (Check  one) 

_  General  Administration 

Nursing  Administration 

_  Dietary 

_  Emergency 

_  Laboratory 

_  Outpatient  Clinic 

_  Outpatient  Nursing 

_  Pharmacy 

_  Radiology 

_  Other  (Specify) : _ 

2.  Job  Description:  (Check  one) 

_  Clerk 

_  Corpsman  (0000) 

_  Technician 

_  Nurse 

_  Pharmacist 

_  Physician 

_  Physician  Assistant 

_  Other  (Specify) : _ 

3.  Highest  Level  of  Education:  (Check  one) 

_  High  School  Graduate 

_  Some  College 

_  Bachelor's  Degree 

_  Some  Graduate  Work 

_  Master's  Degree 

_  Doctoral  Degree 

_  Medical  Degree 

_  Other  (Specify) : _ 

4.  Age:  _  years 

5.  Gender:  _  Male  _  Female 

6.  Length  of  time  (in  months)  you  have  used  CHCS: 

7.  Have  you  used  other  computer  systems  before  ?  _  No  _Jfes 

If  your  answer  was  Yes,  was  it  a  health  care 

information  system  ?  _  Yes  _  No 
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Part  B:  The  Questionnaire 

This  section  of  the  survey  conveys  your  own  personal 
feelings  concerning  the  use  of  the  Composite  Health  Care 
system  at  Naval  Hospital,  (LOCATION).  Please  do  not  attempt 
to  analyze  the  questions.  Remember,  there  are  no  right  or 
wrong  answers. 

Please  follow  these  instructions: 

a.  Check  each  scale  in  the  position  that  describes  your 
evaluation  of  the  factor  being  described. 

b.  Check  each  scale,  do  not  omit  any. 

c.  Check  only  one  position  for  each  scale. 

d.  Check  in  the  space,  not  between  spaces.  THIS,  NOT  THIS 

:  X  :  _ X _ : 

e.  Work  rapidly.  Rely  on  your  first  impressions. 

The  scale  positions  are  defined  as  follows: 

adjective  X  : _ : _ : _ : _ : _ : _ : _ :  adjective  Y 

(1)  (2)  (3)  (4)  (5)  (6)  (7) 


(1) 

extremely  X 

(5)  slightly  Y 

(2) 

quite  X 

(6)  quite  Y 

(3) 

slightly  X 

(7)  extremely  Y 

(4) 

neither  X  or  Y; 

equally  X  or  Y;  does  not  apply 

ANSWERS  BASED  ON  YOUR  OWN  FEELINGS 


1.  Relationship  with  the  Management  information  Department 


(MID)  staff 

dissonant  : _ 

•  •  • 

•  •  • 

•  • 

•  • 

_ : _ :  harmonious 

bad  : _ 

•  •  • 

•  •  • 

:  ; 

:  :  good 

2.  Processing  of  requests  for  changes  to  existing  systems 

fast  : _ : _ : _ : _ : _ : _ : _ :  slow 

untimely  : _ : _ : _ : _ : _ : _ : _ :  timely 

3.  Degree  of  training  provided  to  users 

complete  : _ : _ : _ : _ : _ : _ : _ :  incomplete 

low  : _ : _ : _ : _ : _ : _ : _ :  high 
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4 .  Your  understanding  of  systems 

insufficient  : _ : _ : _ : _ : _ : _ : _ :  sufficient 

complete  : _ : _ : _ : _ : _ : _ : _ :  incomplete 

5.  Your  feeling  of  participation 

positive  : _ : _ : _ : _ : _ : _ : _ :  negative 

insufficient  : _ : _ : _ : _ : _ : _ : _ :  sufficient 

6.  Attitude  of  the  Management  Information  Department  staff 

cooperative  : _ : _ : _ : _ : _ : _ : _ :  belligerent 

negative  : _ : _ : _ : _ : _ : _ : _ :  positive 

7.  Reliability  of  output  information 

high  : _ : _ : _ : _ : _ : _ : _ :  low 

superior  : _ : _ : _ : _ : _ : _ : _ :  inferior 

8.  Relevancy  of  output  information  to  intended  function 
(degree  of  what  user  wants  or  requires  and  what  is  provided  by 
the  system) 

useful  : _ : _ : _ : _ : _ : _ : _ :  useless 

relevant  : _ : _ : _ : _ : _ : _ : _ :  irrelevant 

9.  Accuracy  of  output  information 

inaccurate  : _ : _ : _ : _ : _ : _ : _ :  accurate 

low  : _ : _ : _ : _ : _ : _ : _ :  high 

10.  Precision  of  output  information  (the  variability  of  the 
output  information  from  that  which  it  purports  to  measure) 

low  : _ : _ : _ : _ : _ : _ : _ :  high 

definite  : _ : _ : _ : _ : _ : _ : _ :  uncertain 

11.  Communication  with  the  Management  Information  Department 
staff 

dissonant  : _ : _ _ : _ : _ : _ : _ :  harmonious 

destructive  : _ : _ : _ : _ : _ : _ : _ :  productive 


12 .  Time  required  for  new  systems  development 

unreasonable  : _ : _ : _ : _ : _ : _ : _ :  reasonable 

acceptable  : _ : _ : _ : _ : _ : _ : _ :  unacceptable 

13.  Completeness  of  the  output  information 

sufficient  : _ : _ : _ : _ : _ : _ : _ :  insufficient 

adequate  : _ : _ : _ : _ : _ : _ : _ :  inadequate 

Thank  you  for  your  cooperation 
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APPENDIX  B 


Exploratory  Factor  Analysis  SAS  Commands: 
libname  dataname  "a:\"; 

title  "CHCS  UIS  using  Baroudi  Short-Form  Questionaire" ; 
title2  "CHCS  (3  hospitals  combined)  Factor  Analysis"; 
option  linesize=80; 
options  pagesize=58; 

data  dataname.dat  (keep=Ql  Q2  Q3  Q4  Q5  Q6  Q7  Q8  Q9  Q10  Qll  Q12  Q13) ; 
infile  "A:\FA-CHCS.DAT"; 

input  Q1  Q2  Q3  Q4  Q5  Q6  Q7  Q8  Q9  Q10  Qll  Q12  Q13; 
proc  factor  method=ml 

msa  /*  Kaiser  Measure  of  Sampling  Adequacy  -  Sphericity  */ 

scree 

heywood 

n=l 

rotate=varimax 
reorder ; 

title3  "Maximum  Likeliness  Factor  Analysis  with  One  Factor"; 
proc  factor  method=ml 
heywood 
n=2 

rotate=varimax 
reorder ; 

title3  "Maximum  Likeliness  Factor  Analysis  with  Two  Factors"; 
proc  factor  method=ml 
heywood 
n=3 

rotate=varimax 

reorder; 

title3  "Maximum  Likeliness  Factor  Analysis  with  Three  Factors"; 
proc  factor  method=ml 
heywood 
n=4 

rotate=varimax 

reorder; 

title3  "Maximum  Likeliness  Factor  Analysis  with  Four  Factors"; 
proc  factor  method=ml 
heywood 
n=5 

rotate=varimax 
reorder ; 

title3  "Maximum  Likeliness  Factor  Analysis  with  Five  Factors"; 
proc  standard  mean=0  std=l  out=stdized; 

title3  "CHCS  data  set  Standardized  with  Mean=0  and  STD=1" ; 
proc  factor  method=ml  data=work. stdized 
heywood 
n=3 

rotate=varimax 

reorder; 

title3  "Max-Likeliness  Factor  Analysis  w/3  Factors 
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(Standardized) " ; 

proc  factor  method=ml  data=work. stdized 
heywood 
n=4 

rotate=varimax 
reorder ; 

title3  MMax-Likeliness  Factor  Analysis  w/4 

(Standardized) " ; 

proc  factor  method=ml  data=work. stdized 
heywood 
n=5 

rotate=variroax 
reorder ; 

title3  "Max-Likeliness  Factor  Analysis  w/5 

(Standardized) " ; 
run; 


Factors 


Factors 
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APPENDIX  C 


Confirmatory  Factor  Analysis: 

PRELIS  Commands: 

CHCS  (3  hospitals  combined)  Factor  Analysis  Confirmation  by 
LISREL 
DA  NI=13 
LA 

Q1  Q2  Q3  Q4  Q5  Q6  Q7  Q8  Q9  Q10  Qll  Q12  Q13 
RA  FI=A: \FA-CHCS.DAT 
OR  ALL 

OU  MA=PM  SM=A:\CHCS.PML  SA=A: \CHCS . ACP  PA 
LISREL  Commands: 

FA  of  Ordinal  Variables  Confirmation  by  LISREL:  3  Hosps 
Combined 

DA  NI=13  N0=340  MA=PM 
LA 

Q1  Q2  Q3  Q4  Q5  Q6  Q7  Q8  Q9  Q10  Qll  Q12  Q13 
PM  FI=A : \CHCS . PML 
AC  FI=A : \CHCS . ACP 
MO  NX=13  NK=4  PH=ST 
LK 

FACTOR_A  FACTOR_B  FACTOR_C  FACTOR_D 
FR  LX  l  1  LX  6  l  LX  ll  l  continued 
LX  2  2  LX  12  2  continued 

LX  7  3  LX  8  3  LX  9  3  LX  10  3  LX  13  3  continued 
LX  3  4  LX  4  4  LX  5  4 
OU  SE  TV 
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APPENDIX  D 


Cronbach's  Reliability  Testing  SAS  Commands: 
libname  dataname  "a:\"; 

title  "CHCS  UIS  using  Baroudi  Short-Form  Questionaire" ; 
title2  "CHCS  (3  hospitals  combined)  Factor  Analysis"; 
title3  "Cronbach  alphas  on  factor  clusters" ; 
options  linesize=80; 
options  pagesize=58; 

data  dataname.dat  (keep=Ql  Q2  Q3  Q4  Q5  Q6  Q7  Q8  Q9  Q10  Qll  Q12 

Q13)  ; 

infile  "a:\FA-CHCS.dat"; 

input  Q1  Q2  Q3  Q4  Q5  Q6  Q7  Q8  Q9  Q10  Qll  Q12  Q13; 
proc  corr  alpha; 

var  Q7  Q8  Q9  Q10  Q13; 
proc  corr  alpha; 

var  Q1  Q6  Qll; 
proc  corr  alpha; 

var  Q3  Q4  Q5 ; 
proc  corr  alpha; 
var  Q2  Q12 ; 

run; 
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APPENDIX  E 


Equivalence  Testing  PRELIS/LISREL  Commands: 

PRELIS  Commands: 

PRELIS  covariance  matrix  for  Charleston  Hospital 
DA  NI=13 
LA 

Q1  Q2  Q3  Q4  Q5  Q6  Q7  Q8  Q9  Q10  Qll  Q12  Q13 
RA  FI=A: \FA-CHAR.DAT 
OR  ALL 

OU  MA=CM  SM=A : \CHAR-LIS . CMX 

PRELIS  covariance  matrix  for  Camp  Lejeune  Hospital 
DA  NI=13 
LA 

Q1  Q2  Q3  Q4  Q5  Q6  Q7  Q8  Q9  Q10  Qll  Q12  Q13 
RA  FI=A: \FA-CLEJ.DAT 
OR  ALL 

OU  MA=CM  SM=A: \CLEJ-LIS . CMX 

PRELIS  covariance  matrix  for  Jacksonville  Hospital 
DA  NI=13 
LA 

Q1  Q2  Q3  Q4  Q5  Q6  Q7  Q8  Q9  Q10  Qll  Q12  Q13 
RA  FI=A: \FA-JAX.DAT 
OR  ALL 

OU  MA=CM  SM=A : \JAX-LIS . CMX 
LISREL  Commands: 

HYPOTHESIS  A,  LOCATION:  NH  CHARLESTON 
DA  NG=3  NI=13  N0=101 
LA 

Q1  Q2  Q3  Q4  Q5  Q6  Q7  Q8  Q9  Q10  Qll  Q12  Q13 
CM  FI=A : \CHAR-LIS . CMX 
MO  NX=13  NK=13  LX=ID  TD=ZE 
OU 

HYPOTHESIS  A,  LOCATION:  NH  CAMP  LEJEUNE 
DA  NO=121 
LA 

Q1  Q2  Q3  Q4  Q5  Q6  Q7  Q8  Q9  Q10  Qll  Q12  Q13 
CM  FI=A : \CLEJ-LIS • CMX 
MO  PH=IN 
OU 

HYPOTHESIS  A,  LOCATION:  NH  JACKSONVILLE 
DA  NO=118 
LA 

Q1  Q2  Q3  Q4  Q5  Q6  Q7  Q8  Q9  Q10  Qll  Q12  Q13 
CM  FI=A : \JAX-LIS . CMX 
MO  PH=IN 
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ou 


HYPOTHESIS  B,  LOCATION:  NH  CHARLESTON 
DA  NG=3  NI=13  N0=101 
LA 

Q1  Q2  Q3  Q4  Q5  Q6  Q7  Q8  Q9  Q10  Qll  Q12  Q13 
CM  FI=A : \CHAR-LIS . CMX 
MO  NX=13  NK=4 

FR  LX  8  1  LX  9  1  LX  10  1  LX  13  1  continued 
LX  2  2  continued 
LX  1  3  LX  6  3  continued 
LX  3  4  LX  4  4 

VA  1  LX  7  1  LX  12  2  LX  11  3  LX  5  4 
OU 

HYPOTHESIS  B,  LOCATION:  NH  CAMP  LEJEUNE 
DA  NO=121 
LA 

Q1  Q2  Q3  Q4  Q5  Q6  Q7  Q8  Q9  Q10  Qll  Q12  Q13 
CM  FI=A: \CLEJ-LIS . CMX 
MO  LX=PS 
OU 

HYPOTHESIS  B,  LOCATION:  NH  JACKSONVILLE 
DA  NO=118 
LA 

Q1  Q2  Q3  Q4  Q5  Q6  Q7  Q8  Q9  Q10  Qll  Q12  Q13 
CM  FI=A: \JAX-LIS . CMX 
MO  LX=PS 
OU 

HYPOTHESIS  C,  LOCATION:  NH  CHARLESTON 
DA  NG=3  NI=13  N0=101 
LA 

Q1  Q2  Q3  Q4  Q5  Q6  Q7  Q8  Q9  Q10  Qll  Q12  Q13 
CM  FI=A : \CHAR-LIS . CMX 
MO  NX=13  NK=4 

FR  LX  8  1  LX  9  1  LX  10  1  LX  13  1  continued 
LX  2  2  continued 
LX  1  3  LX  6  3  continued 
LX  3  4  LX  4  4 

VA  1  LX  7  1  LX  12  2  LX  11  3  LX  5  4 
OU 

HYPOTHESIS  C,  NH  CAMP  LEJEUNE 
DA  NO=121 
LA 

Q1  Q2  Q3  Q4  Q5  Q6  Q7  Q8  Q9  Q10  Qll  Q12  Q13 
CM  FI=A : \CLEJ-LIS . CMX 
MO  LX=IN 
OU 

HYPOTHESIS  C,  LOCATION:  NH  JACKSONVILLE 
DA  NO=118 
LA 

Q1  Q2  Q3  Q4  Q5  Q6  Q7  Q8  Q9  Q10  Qll  Q12  Q13 
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CM  FI=A: \JAX-LIS . CMX 
MO  LX=IN 
OU 

HYPOTHESIS  D,  LOCATION:  NH  CHARLESTON 
DA  NG=3  NI=13  N0=101 
LA 

Q1  Q2  Q3  Q4  Q5  Q6  Q7  Q8  Q9  Q10  Qll  Q12  Q13 
CM  FI=A: \CHAR-LIS . CMX 
MO  NX=13  NK=4 

FR  LX  8  1  LX  9  1  LX  10  1  LX  13  1  continued 
LX  2  2  continued 
LX  1  3  LX  6  3  continued 
LX  3  4  LX  4  4 

VA  1  LX  7  1  LX  12  2  LX  11  3  LX  5  4 
OU 

HYPOTHESIS  D,  LOCATION:  NH  CAMP  LEJEUNE 
DA  NO=121 
LA 

Q1  Q2  Q3  Q4  Q5  Q6  Q7  Q8  Q9  Q10  Qll  Q12  Q13 
CM  FI=A: \CLEJ-LIS . CMX 
MO  LX=IN  TD=IN 
OU 

HYPOTHESIS  D,  LOCATION:  NH  JACKSONVILLE 
DA  NO=118 
LA 

Q1  Q2  Q3  Q4  Q5  Q6  Q7  Q8  Q9  Q10  Qll  Q12  Q13 
CM  FI=A: \JAX-LIS . CMX 
MO  LX=IN  TD=IN 
OU 

HYPOTHESIS  E,  LOCATION:  NH  CHARLESTON 
DA  NG=3  NI-13  N0=101 
LA 

Q1  Q2  Q3  Q4  Q5  Q6  Q7  Q8  Q9  Q10  Qll  Q12  Q13 
CM  FI=A: \CHAR-LIS . CMX 
MO  NX=13  NK=4 

FR  LX  8  1  LX  9  1  LX  10  1  LX  13  1  continued 
LX  2  2  continued 
LX  1  3  LX  6  3  continued 
LX  3  4  LX  4  4 

VA  1  LX  7  1  LX  12  2  LX  11  3  LX  5  4 
OU 

HYPOTHESIS  E,  LOCATION:  NH  CAMP  LEJEUNE 
DA  NO=121 
LA 

Q1  Q2  Q3  Q4  Q5  Q6  Q7  Q8  Q9  Q10  Qll  Q12  Q13 
CM  FI=A: \CLEJ-LIS . CMX 
MO  LX=IN  TD=IN  PH=IN 
OU 

HYPOTHESIS  E,  LOCATION:  NH  JACKSONVILLE 
DA  NO=118 
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LA 

Q1  Q2  Q3  Q4  Q5  Q6  Q7  Q8  Q9  Q10  Qll  Q12  Q13 
CM  FI=A: \JAX-LIS . CMX 
MO  LX=IN  TD=IN  PH=IN 
OU 
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APPENDIX  F 


CHCS  -  COMBINED  3  HOSPITAL  DATA 
SHORT-FORM  UIS  INSTRUMENT  OVERALL  STATISTICS 


TOTAL  NUMBER  OF  SURVEYS: 
RESPONSE  RATE  (680  SENT) 
MEDIAN  EDUCATION  LEVEL: 
AVERAGE  AGE: 

MINIMUM  AGE: 

MAXIMUM  AGE: 

NUMBER  OF  MALES: 

NUMBER  OF  FEMALES: 

PERCENT  MALES : 

PERCENT  FEMALES: 

AVERAGE  MONTHS  USED: 

MIN  MONTHS  USED: 

MAX  MONTHS  USED: 

NUMBER  USED  COMPUTER  BEFORE 
PERCENT  USED  BEFORE: 

NUMBER  USED  HLT  CARE  COMPTR 
PERCENT  USED  HC  COMP  BEFORE 
PERCENT  USED  BEFORE  AND 
USED  HC  SYSTEM  BEFORE 


340 

50% 

1  Some  College 
32  years 
18  years 
61  years 
213 
127 
63% 

37% 

12.3 

1 

36 

271 

80% 

200 

59% 

74% 


NO.  PERSN 

MONTHS 

USE 

1-5: 

67 

20% 

NO.  PERSN 

MONTHS 

USE 

6-11: 

98 

29% 

NO.  PERSN 

MONTHS 

USE 

12-17: 

81 

24% 

NO.  PERSN 

MONTHS 

USE 

18  >: 

94 

28% 

QUEST  1 

QUEST  2 

QUEST  3 

QUEST  4 

QUEST  5 

QUEST  6 

AVERAGE 

1.27 

-0.41 

0.15 

0.72 

0.68 

1.43 

STD  DEV 

1.29 

1.62 

1.73 

1.68 

1.57 

1.30 

MIN  NUMBR 

-3 

-3 

-3 

-3 

-3 

-3 

MAX  NUMBR 

3 

3 

3 

3 

3 

3 

QUEST  7 

QUEST  8 

QUEST  9 

QUEST  10 

QUEST  11 

QUEST  12 

AVERAGE 

1.00 

0.93 

1.16 

0.75 

1.14 

-0.23 

STD  DEV 

1.42 

1.41 

1.38 

1.32 

1.36 

1.67 

MIN  NUMBR 

-3 

-3 

-3 

-3 

-3 

-3 

MAX  NUMBR 

3 

3 

3 

3 

3 

3 

QUEST  13  OVERALL 


AVERAGE  0.69  9.26 

STD  DEV  1.53  12.77 

MIN  NUMBR  -3  -28 

MAX  NUMBR  3  39 
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MID  CONTRACTOR  INFORMATION  KNOWLEDGE  & 

STAFF/SERV  STAFF/ SERV  PRODUCT  INVOLVEMENT 

FACTOR  A  FACTOR  B  FACTOR  C  FACTOR  D 

(1,6,11)  (2,12)  (7,8,9,10,13)  (3,4,5) 

AVERAGE  1.28  -0.32  0.91  0.51 

STD  DEV  1.19  1.43  1.15  1.35 

MIN  NUMBR  -3  -3  -3  -3 

MAX  NUMBR  33  33 


NAVAL  HOSPITAL  CHARLESTON,  SC 
CHCS ,  SHORT-FORM  UIS  INSTRUMENT  STATISTICS 


TOTAL  NUMBER  OF  SURVEYS:  101 

RESPONSE  RATE  (180  SENT):  56% 

MEDIAN  EDUCATION  LEVEL:  Some  college 


AVERAGE  AGE: 

32 

years 

MINIMUM  AGE: 

19 

years 

MAXIMUM  AGE: 

56 

years 

NUMBER  OF  MALES:  (Percent) 

57 

(56%) 

NUMBER  OF  FEMALES: 

44 

(44%) 

AVERAGE  MONTHS  USED: 

8.6 

MIN  MONTHS  USED: 

1 

MAX  MONTHS  USED: 

19 

NUMBER  USED  COMPUTER  BEFORE 

83 

PERCENT  USED  BEFORE: 

82% 

NUMBER  USED  HLT  CARE  COMPTR 

59 

PERCENT  USED  HC  COMP  BEFORE 

58% 

PERCENT  USED  COMP  BEFORE  AND 

USED  HC  SYSTEM  BEFORE: 

71% 

NO.  PERSN  MONTHS  USE  1-5: 

28 

28% 

NO.  PERSN  MONTHS  USE  6-11: 

36 

36% 

NO.  PERSN  MONTHS  USE  12  >: 

37 

36% 

QUEST  1  QUEST  2 

QUEST  3 

QUEST  4 

QUEST  5 

QUEST  6 

AVERAGE  1.68  -0.50 

0.57 

0.97 

1.07 

1.99 

STD  DEV  1.22  1.67 

1.72 

1.60 

1.57 

1.00 

MIN  NUMBR  -2  -3 

-3 

-3 

-3 

-1.5 

MAX  NUMBR  3  3 

3 

3 

3 

3 

QUEST  7  QUEST  8 

QUEST  9 

QUEST  10 

QUEST  11 

QUEST  12 

AVERAGE  0.79  0.96 

1.17 

0.54 

1.65 

-0.22 

STD  DEV  1.47  1.39 

1.50 

1.43 

1.14 

1.64 

MIN  NUMBR  -3  -3 

-3 

-3 

-0.5 

-3 

MAX  NUMBR  3  3 

3 

3 

3 

3 

QUEST  13  OVERALL 

AVERAGE  0.59  11.26 

STD  DEV  1.44  11.80 

MIN  NUMBR  -3  -13.5 

MAX  NUMBR  3  39 
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MID  CONTRACTOR  INFORMATION  KNOWLEDGE  & 

STAFF/ SERV  STAFF/SERV  PRODUCT  INVOLVEMENT 

FACTOR  A  FACTOR  B  FACTOR  C  FACTOR  D 

(1,6,11)  (2,12)  (7,8,9,10,13)  (3,4,5) 

AVERAGE  1.77  -0.36  0.81  0.87 

STD  DEV  0.99  1.34  1.17  1.25 

MIN  NUMBR  -1  -3  -3  -2 

MAX  NUMBR  333  3 


NAVAL  HOSPITAL  CAMP  LEJEUNE ,  NC 
CHCS ,  SHORT-FORM  UIS  INSTRUMENT  STATISTICS 


TOTAL  NUMBER  OF  SURVEYS: 

121 

RESPONSE  RATE  (250  SENT) : 

48% 

MEDIAN  EDUCATION  LEVEL: 

Some  college 

AVERAGE  AGE: 

33 

years 

MINIMUM  AGE: 

18 

years 

MAXIMUM  AGE: 

61 

years 

NUMBER  OF  MALES:  (Percent) 

73 

(60%) 

NUMBER  OF  FEMALES:  (Percent) 

48 

(40%) 

AVERAGE  MONTHS  USED: 

12.5 

MIN  MONTHS  USED: 

1 

MAX  MONTHS  USED: 

36 

NUMBER  USED  COMPUTER  BEFORE 

92 

PERCENT  USED  BEFORE: 

76% 

NUMBER  USED  HLT  CARE  COMPTR 

63 

PERCENT  USED  HC  COMP  BEFORE 

52% 

PERCENT  USED  BEFORE  AND 

USED  HC  SYSTEM  BEFORE 

68% 

NO.  PERSN  MONTHS  USE  1-5: 

21 

17% 

NO.  PERSN  MONTHS  USE  6-11: 

44 

37% 

NO.  PERSN  MONTHS  USE  12-17: 

27 

27% 

NO.  PERSN  MONTHS  USE  18  >: 

29 

29% 

QUEST  1 

QUEST  2 

QUEST  3 

QUEST  4 

QUEST  5 

QUEST  6 

AVERAGE 

1.44 

0.19 

0.44 

0.67 

0.84 

1.63 

STD  DEV 

1.27 

1.41 

1.68 

1.82 

1.48 

1.30 

MIN  NUMBR 

-2 

-3 

-3 

-3 

-3 

-1.5 

MAX  NUMBR 

3 

3 

3 

3 

3 

3 

QUEST  7 

QUEST  8 

QUEST  9 

QUEST  10 

QUEST  11 

QUEST  12 

AVERAGE 

1.28 

1.16 

1.38 

0.86 

1.31 

0.14 

STD  DEV 

1.29 

1.35 

1.26 

1.29 

1.39 

1.57 

MIN  NUMBR 

-3 

-3 

-3 

-3 

-3 

-3 

MAX  NUMBR 

3 

3 

3 

3 

3 

3 
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QUEST  13  OVERALL 


AVERAGE 

0.94 

12.28 

STD  DEV 

1.46 

12.41 

MIN  NUMBR 

-3 

-25 

MAX  NUMBR 

3 

39 

MID 

CONTRACTOR 

INFORMATION 

KNOWLEDGE  & 

STAFF/SERV 

STAFF/SERV 

PRODUCT 

INVOLVEMENT 

FACTOR  A 

FACTOR  B 

FACTOR  C 

FACTOR  D 

(1,6,11) 

(2,12) 

(7,8,9,10,13) 

(3,4,5) 

AVERAGE  1.46 

0.17 

1.12 

0.65 

STD  DEV  1 . 17 

1.32 

1.07 

1.38 

MIN  NUMBR  -1 

-3 

-3 

-3 

MAX  NUMBR  3 

3 

3 

3 

NAVAL  HOSPITAL  JACKSONVILLE,  FL 
CHCS ,  SHORT-FORM  UIS  INSTRUMENT  STATISTICS 


TOTAL  NUMBER  OF  SURVEYS: 
RESPONSE  RATE  (250  SENT) : 
MEDIAN  EDUCATION  LEVEL: 
AVERAGE  AGE: 

MINIMUM  AGE: 

MAXIMUM  AGE: 

NUMBER  OF  MALES:  (Percent) 
NUMBER  OF  FEMALES:  (Percent) 
AVERAGE  MONTHS  USED: 

MIN  MONTHS  USED: 

MAX  MONTHS  USED: 

NUMBER  USED  COMPUTER  BEFORE 
PERCENT  USED  BEFORE: 


118 

47% 

BACHELOR  DEGREE 
32  YEARS 
19  YEARS 
56  YEARS 
83  (70%) 

35  (30%) 

15.3 

1 

32 

102 

86% 


NUMBER  USED  HLT  CARE  COMPTR  78 
PERCENT  USED  HC  COMP  BEFORE  66% 
PERCENT  USED  BEFORE  AND 

USED  HC  SYSTEM  BEFORE  76% 
NO.  PERSN  MONTHS  USE  1-5:  18 
NO.  PERSN  MONTHS  USE  6-11:  18 
NO.  PERSN  MONTHS  USE  12-17:  22 
NO.  PERSN  MONTHS  USE  18  >:  60 


15% 

15% 

19% 

51% 


QUEST  1 

QUEST  2 

QUEST  3 

QUEST  4 

QUEST  5 

AVERAGE 

0.74 

-0.94 

-0.53 

0.56 

0.17 

STD  DEV 

1.22 

1.57 

1.60 

1.58 

1.55 

MIN  NUMBR 

-3 

-3 

-3 

-3 

-3 

MAX  NUMBR 

3 

3 

3 

3 

3 

QUEST  6 
0.74 
1.23 
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u>  w 


QUEST  7 

QUEST  8  QUEST 

9 

QUEST  10  QUEST  11  QUEST 

AVERAGE 

0.89 

0.67  0. 

94 

0.81 

0.52  -0 

STD  DEV 

1.47 

1.47  1. 

36 

1.22 

1.25  1 

MIN  NUMBR 

-3 

-3 

-3 

-3 

-3 

MAX  NUMBR 

3 

3 

3 

3 

3 

QUEST  13 

OVERALL 

AVERAGE 

0.52 

4.46 

STD  DEV 

1.66 

12.63 

MIN  NUMBR 

-3 

-28 

MAX  NUMBR 

3 

36 

MID 

CONTRACTOR 

INFORMATION 

KNOWLEDGE  & 

STAFF/SERV  STAFF/SERV 

PRODUCT 

INVOLVEMENT 

FACTOR  A 

FACTOR  B 

FACTOR  C 

FACTOR  D 

(1,6,11) 

(2,12) 

(7, 

8,9,10,13) 

(3,4,5) 

AVERAGE 

0.67 

-0.79 

0.77 

0.07 

STD  DEV 

1.13 

1.47 

1.19 

1.30 

MIN  NUMBR 

-3 

-3 

-3 

-3 

MAX  NUMBR 

3 

3 

3 

3 

107 


APPENDIX  G 

Part  A:  General  Information 


1.  Hospital  Department:  (Check  one) 

_  General  Administration 

_  Nursing  Administration 

_  Dietary 

_  Emergency 

_  Laboratory 

_  Outpatient  Clinic 

_  Outpatient  Nursing 

_  Pharmacy 

_  Radiology 

_  Other  (Specify) : _ 

2.  Job  Description:  (Check  one) 

_  Clerk 

_  Corpsman  (0000) 

_  Technician 

_  Nurse 

_  Pharmacist 

_  Physician 

_  Physician  Assistant 

_  Other  (Specify) : _ 

3.  Length  of  time  (in  months)  you  have  used  CHCS: 


Part  B:  The  Questionnaire 

This  section  of  the  survey  conveys  your  own  personal  feelings 
concerning  the  use  of  the  Composite  Health  Care  system  at  Naval 
Hospital,  (LOCATION) .  Please  do  not  attempt  to  analyze  the  questions. 
Remember,  there  are  no  right  or  wrong  answers. 

Please  follow  these  instructions: 

a.  Check  each  scale  in  the  position  that  describes  your  evaluation 
of  the  factor  being  described. 

b.  Check  each  scale,  do  not  omit  any. 

c.  Check  only  one  position  for  each  scale.  THIS,  NOT  THIS 

d.  Check  in  the  space,  not  between  spaces.  :  X  : _ X _ : 

e.  Work  rapidly.  Rely  on  your  first  impressions. 

The  scale  positions  are  defined  as  follows: 

adjective  X  : _ : _ : _ : _ : _ : _ :  adjective  Y 

(1)  (2)  (3)  (4)  (5)  (6) 


(1) 

extremely  X 

(4) 

slightly  Y 

(2) 

quite  X 

(5) 

quite  Y 

(3) 

slightly  X 

(6) 

extremely  Y 

The  following  example  illustrates  the  scale  positions  and  their 
meanings: 

My  vacation  in  the  Bahamas  was: 

restful  : _ : _ : _ : _ : _ :  X  :  hectic 

healthy  : _ :  X  : _ : _ : _ : _ :  unhealthy 

According  to  the  responses,  the  person's  vacation  was  extremely  hectic 
and  quite  healthy 

ANSWERS  BASED  ON  YOUR  OWN  FEELINGS 

1.  Relationship  with  the  Management  Information  Department  (MID) 
staff 

dissonant  : _ : _ : _ : _ : _ : _ :  harmonious 

bad  : _ : _ : _ : _ : _ : _ :  good 

2 .  Processing  of  requests  for  changes  to  existing  system 

fast  : _ : _ : _ : _ : _ : _ :  slow 

untimely  : _ : _ : _ : _ : _ : _ :  timely 

3 .  Degree  of  training  provided  to  you 

complete  : _ : _ : _ : _ _ : _ :  incomplete 

low  : _ : _ : _ : _ : _ : _ :  high 

4 .  Your  understanding  of  the  system 

insufficient  : _ : _ : _ : _ : _ : _ :  sufficient 

complete  : _ : _ : _ : _ : _ : _ :  incomplete 

5.  Your  feeling  of  participation 

positive  : _ : _ : _ : _ : _ : _ :  negative 

insufficient  : _ : _ : _ : _ : _ : _ :  sufficient 

6.  Attitude  of  the  Management  Information  Department  staff 

cooperative  : _ : _ : _ : _ : _ : _ :  belligerent 

negative  : _ : _ : _ : _ : _ : _ :  positive 
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7.  Reliability  of  output  information  (that  the  output  is  reliable, 
timely,  accurate,  and  precise) 

high  : _ : _ : _ : _ : _ : _ :  low 

superior  : _ : _ : _ : _ : _ : _ :  inferior 

8.  Vendor  Support  (the  type  and  quality  of  the  service  rendered  by 
the  vendor) 

sufficient  : _ : _ : _ : _ : _ : _ : _ :  insufficient 

poor  : _ : _ : _ : _ : _ : _ : _ :  good 

9.  Convenience  of  access  (the  ease  or  difficulty  that  you  have  to 
utilize  the  capability  of  the  system) 

easy  : _ : _ : _ : _ : _ : _ : _ :  difficult 

efficient  : _ : _ : _ : _ : _ : _ : _ :  inefficient 

10.  Communication  with  the  Management  Information  Department  staff 

dissonant  : _ : _ : _ : _ : _ : _ : _ :  harmonious 

destructive  : _ : _ : _ : _ : _ : _ : _ :  productive 

11.  Time  required  for  new  system  development 

unreasonable  : _ : _ : _ : _ : _ : _ : _ :  reasonable 

acceptable  : _ : _ : _ : _ : _ : _ : _ :  unacceptable 

12 .  Completeness  of  the  output  information 

sufficient  : _ : _ : _ : _ : _ : _ : _ :  insufficient 

adequate  : _ : _ : _ : _ : _ : _ : _ :  inadequate 

Thanks  again  for  your  cooperation 
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